Toward Efficient and Effective Software Sustainment

In my preceding blog posts, I promised to provide more examples highlighting the importance of software sustainment in the U.S. Department of Defense (DoD). My focus is on sustaining legacy weapons systems that are no longer in production, but are expected to remain a key component of our defense capability for decades to come. Despite the fact that these legacy systems are no longer in the acquisition phase, software upgrade cycles are needed to refresh their capabilities every 18 to 24 months. In addition, significant modernization can often be made by more extensive, focused software upgrades with relatively modest hardware changes. This third blog post describes effective sustainment engineering efforts in the Army, using examples from across its software engineering centers. These examples are tied to SEI research on capability maturity models, agility, and the Architecture Analysis and Design Language (AADL) modeling notation.

A Brief History of Software Sustainment in the Army

From its earliest days, the military has provided facilities to maintain the functionality of its various weapon systems. The descriptive terms for these units have included "arsenals," "depots," and "engineering centers," to name a few. In the Army, responsibility is in software centers within the Research, Development, and Engineering Command (RDECOM). Sites are spread across the country, from New Jersey (Picatinny Arsenal) and Maryland (Aberdeen Proving Ground) to Arizona (Fort Huachuca).

Within this geographical framework of arsenals and depots across the country, we can overlay the increased importance of software in our weapon systems. The Abrams Main Battle Tank provides an excellent example. Design began in 1972, with production in 1980. Current predictions are that the latest version (M1A2) will remain in use another 30 years, emphasizing the need for software updates to legacy weapon systems. Similarly, the Army's Apache helicopter has experienced remarkable, software-intense upgrades over its life since its initial design in the 1970s. The initial version, the AH-64A, had about 300,000 source lines of code (KSLOC) aboard. The AH-64D, produced more recently, has over 1.4 million source lines of code (MSLOC), vastly expanding the capabilities of the system. The recently released AH-64E continues the growth of software importance in this mainstay of Army aviation (more about this weapon system later in this posting).

Given the growing importance of software for Army systems, this blog post highlights some recent examples of effective sustainment of software-reliant systems across RDECOM. I focus on the results of improvement efforts across the domain. The workforce in the software centers has grown in its technical competence, and a modern systems engineering environment has been developed. With that historical perspective, we can explore some of the examples evident today.

Continuous Improvement

The Armament Software Engineering Center (Armament SEC) at Picatinny Arsenal has long been involved with SEI technologies. In keeping with the concept that reaching maturity levels using models like CMMI only sets the stage for greater improvements, the Armament SEC team at the arsenal has shared its approach to defect containment with their REDCOM peers. The chart below shows the team's Defect Containment Matrix (DCM), which provides a way of visually displaying information about the issues and problem reports associated with a system.

Defect Containment Matrix Example

Source: ARDEC Armament Software Engineering Center (Picatinny Arsenal)

By using a DCM method, the Armament SEC team can focus its improvement efforts on the areas that most need it. This matrix enables development teams to improve their focus on removing defects so they don't leak into the next phase of the lifecycle. This process allows earlier corrections and saves rework costs.

The Defect Leakage Improvement chart below shows the benefit to the Armament SEC defect removal process from numerous process improvements, such as enhanced software peer reviews. Metrics captured across the improvement cycle, from when the center had been performing at maturity level 3 to its current status at maturity level 5, tell the story.

Defect Leakage Improvement

Source: ARDEC Armament Software Engineering Center (Picatinny Arsenal)

DCM was introduced when The U.S. Army Armament Research, Development and Engineering Center (ARDEC) was at maturity level 3 in CMMI and beginning its measurement of improvement activities, one of which was DCM. When it achieved CMMI maturity level 5 several years later, it measured the difference and discovered the improved containment and reduced leakage shown in the chart above and outlined below:

• Defects captured within a phase increased by more than three times; defect leakage for one phase decreased by more than 11 times: and defect leakage for more than one phase decreased by more than 6 times.
• Less rework occurred late in the lifecycle when it is most expensive to repair, resulting in reduced cost and schedule risks.
• Peer reviews became more effective in attacking systemic issues. Peer review return on investment ~5.5.

Efficient Agile Updates

At Redstone Arsenal in Huntsville, AL the Software Engineering Directorate (SED) initiated a major restructuring of the launcher capabilities for the Precision Fires Rocket and Missile System (PFRMS). Over the years of capability improvement on the launcher, the number of computer system configuration items (CSCIs) had grown to 36, as it was frequently easier to focus on a new segment than to restructure. The software size reached about 1.2 MSLOC. The Huntsville team took on a re-architecting of the system to eliminate redundant elements into 6 CSCI's, and a reduction to about 450 KLOC, about one-third its former size. The restructure significantly shortened maintenance checks--a significant value delivered to the warfighter in the field. While topics like software reliability are nicely covered in other blogs, it is important to note that the code clean-up reduces the likelihood of software deficiencies in the field.

The team tackling the restructure was also leading the transition into agile development, creating new metrics that better describe developmental progress when using agile methods. The cost-effective success achieved using agile methods at SED has led the Army program office to assign more software sustainment efforts for the launcher to SED.

Architecture Analysis and Design Language (AADL)

A number of years ago, the SEI teamed with SED in an effort to improve software-reliant systems through virtual system integration with the Architecture Analysis and Design Language (AADL), which is the industry standard modeling language that was standardized by SAE International. As discussed extensively in a series of SEI blog posts, AADL provides notations for capturing legacy software and system architectures and then evaluating the effects of change through predictive, quantitative analysis. Analysis domains of concern is critical to software-reliant systems and includes aspects (such as hardware utilization, system timing, latency effects, scheduling, fault propagation, fault tolerance, safety, and security) that can only be understood from an architectural perspective.

With the SED at Redstone Arsenal, we again see a sustainment-focused organization involved in turning leading-edge research into practical solutions for problems faced when upgrading and evolving legacy weapons systems. By applying AADL to model alternative architectures and interface options for new avionics in aircraft like the Apache AH-46E mentioned above, SED can explore and verify safer and more efficient upgrades to the weapon systems' capabilities with the prime contractor. SED can then evaluate the effect of changes proposed by the prime contractor for the program office. This review may result in an improved approach as options are evaluated or result in the early discovery of system issues. SED and the SEI are also involved in advanced system development experiments for new aircraft systems as a result of this field research with existing systems.

For example, failure modes and effects analysis are far easier when an advanced design language like AADL provides visibility into the ways that errors can propagate across the system of systems that composes a modern aircraft. Similarly, SED can enhance the security of the weapon system by using AADL tools to investigate security vulnerabilities and avoid them in the planned upgrade. An earlier blog post describes how to model system behavior with AADL. Another post discusses the use of the language to capture safety and reliability aspects. To read more about other applications of AADL, please visit our blog landing page and click on the AADL category.

Agile Systems Engineering

Another project at SED demonstrates the value of effective integration of improved methodologies for both hardware and software. The directorate had developed expertise in software design that led to its designation as the center to develop and maintain the game-based training of young soldiers--now called the "America's Army" product suite.

The directorate learned of a need to develop a different hardware and software system--a simulator to train deploying troops on successful operations for each crew station in a mine resistant ambush protected (MRAP) vehicle to include egress training after a rollover. These vehicles had provided vital improvement in the safety of the troops inside it compared with earlier vehicles. However, their design also increased the possibility of a rollover, so egress training--to allow escape from an overturned vehicle--was needed.

I used the term "agile systems engineering" to connote the innovative approach taken to create a solution to this training challenge. The SED project coupled its game capabilities with some hardware collaborators who were part of the SED team. Within 18 months, it developed a prototype of the system within the time normally spent trying to get the requirements written. All the prototype development was done with frequent user involvement, so the key principles of agile development were being used. Moreover, the combination of hardware and software development meant that systems engineering agility was being powerfully demonstrated.

Conclusions and Looking Ahead

For more than 200 years, sustainment efforts across our military systems meant restoring them to an acceptable state for reuse. As the capabilities have become increasingly software-reliant, however, the sustainment segment of the system lifecycle has changed dramatically. The organizations committed to modernizing our military capabilities have proved their expertise and their ability to innovate with techniques and tools freshly out of the research domain. Future blogs will look at other examples from similar SEI research efforts.

Additional Resources

To read the SEI technical report, Sustaining Software Intensive Systems, please visit https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=7865