Quality is something which is hard to define. Different areas of expertise use their own definitions of what quality is. Without an objective standard which carries weight, anyone can claim to provide a quality product or service according to some standard. This makes it difficult to compare products and to formalize which characteristics a product or service needs to have. In this blog post I'll provide an introduction to ISO/IEC-5055 which is a quality standard which allows us to measure the quality of source code objectively.
Standards organizations
The ISO is an international standards organization which has the authority to create standards which carry weight. The members of ISO are national standards institutes. ISO itself is not government owned and independent. The standards ISO creates however cover a large number of areas. Specifically for electrical, electronic and information technologies, they often work together with the IEC in a Joint Technical Committee (JTC). The IEC also creates standards and they don't want their standards to overlap or compete with each other. By working together in a JTC, they can prevent that.
Part of the standards related to quality have been proposed by the CISQ (Consortium for IT Software Quality). CISQ is a consortium founded by, among others the OMG (the Object Management Group). The OMG is known for standards like CORBA, UML and BPMN. The members of CISQ are private sector companies. Mostly companies who have interest in a shared definition of what quality is. The OMG as part of CISQ managed to get the proposed quality standards through to be promoted to ISO/IEC standards.
What is quality?
The term quality is defined in ISO 9000 (criteria for a quality management system) as the degree to which a set of inherent characteristics of an object fulfills requirements.
Note that:
- “degree” implies quality is a variable
- “a set of” implies quality is not a single characteristic
- “inherent” as opposed to "assigned", means existing in the object therefore neither price nor delivery is a quality characteristic of a product but can be a characteristic of a service
- “object” means anything perceivable or conceivable therefore the term quality can be used relative to both tangible and intangible objects e.g. the quality of commitment.
- “requirement” means a need or expectation that is stated, generally implied or obligatory
There are a lot of reasons why measuring quality can be beneficial. If you measure quality in an objective manner, you are able to compare products, modules and by proxy, the people or companies producing them. You can apply quality standards directly in contracts. You can use quality measures as a way to promote your company and products. Measuring quality can help prevent weaknesses and maybe even prevent product liability cases.
For developers, it helps to know if you make choices in your code which can lead to lower quality (think performance, security, stability, maintainability). If you do not measure it, you could be unaware if you make wrong choices. Visualizing code quality is also a great motivator. You can try and become a better programmer by providing measurable better code.
ISO/IEC 25010
ISO/IEC 25010 defines a standard for evaluating system and software product quality. Part of this standard are the definition of quality characteristics.
- The measures are not compatible with each other and cannot be used to obtain a single number indicating quality. For example, part of reliability is availability. This is measured by mean down time. Another aspect of reliability is recoverability. Recoverability is measured by mean recovery time. These two measures (together with the other aspects of reliability) cannot be summed to create a single measure for reliability.
- The measures are mostly behavioral. They are measures which apply to a running application which is in use and not so much to the source code.
ISO/IEC 5055 complies to ISO/IEC 25010 and expands ISO/IEC 25023 by defining measures which are applicable to source code. It is a freely available standard which you can download here. It has been published in April 2021 and is thus a relatively recent standard.
An important reasons why ISO-5055 was created is IoT. Software is used by IoT devices usually contain code which is difficult to update once weaknesses are discovered. For example updating the device requires manual actions at the location of the device. Since it is expensive to fix issues (especially in IoT cases), it helps if you can detect quality issues early during development, preferable even in a development environment, before official tests have been conducted or a release is made. This saves a lot of time, effort and thus costs.
ISO/IEC 5055 defines weaknesses and detection patterns which allow automation. The weaknesses themselves are registered in the CWE (Common Weakness Enumeration) database which is maintained by MITRE. MITRE is the same organization who also maintain the CVE (Common Vulnerabilities and Exposures) database. Usually CVEs can be linked to CWEs. CWEs are usually less specific (often do not specify a concrete product or framework). Initially the CWE database only contained security related information but this has been expanded to also contain reliability, performance and maintainability issues. You can browse the ISO/IEC-5055 weaknesses here.
The weaknesses which are described in ISO/IEC-5055 have been evaluated by an international team of experts and identified as requiring remediation. ISO/IEC-5055 contains 138 weaknesses. Because it is a relatively short list, it can relatively easily be automated.
The weaknesses cover different layers of an architecture. From unit to layer to system. An important reason for this is that applications nowadays, mostly cannot be looked at in isolation as a single black box but are part of a larger network and also have an internal structure.
Below are some examples of measures at different levels;
Tools
While ISO/IEC 5055 provides a definition of weaknesses and detection patterns, but manually checking them in for example a code review, is inaccurate and boring. Since the detection patterns are formalized, they can be automated. Since ISO/IEC 5055 is a relatively new standard, not many tools are currently compliant, but that number is expected to grow. The Executive Director of CISQ is also Chief Scientist at CAST (Dr. Bill Curtis). CAST 'MRI for Software 'is the first tool which is claimed to be compliant (here).
Gartner is a research and advisory company which provides (among others) comparisons of software in so-called magic quadrants. Below you can see the magic quadrant of the 'Application Security Testing' category of the past 3 years. This category overlaps with the weaknesses which are described in ISO/IEC 5055 (completely with the security weaknesses and partially with the other weaknesses).
- Synopsys (29%)
- Veracode (18%)
- Micro Focus (8%)
- The ISO/IEC 5055 standard
https://standards.iso.org/ittf/PubliclyAvailableStandards/c080623_ISO_IEC_5055_2021(E).zip - CISQ webinar on ISO/IEC 5055
https://www.it-cisq.org/webinars/iso-5055-automated-source-code-quality-measures.htm - CISQ ISO/IEC 5055 weaknesses as documented by MITRE
https://cwe.mitre.org/data/definitions/1305.html - CWE coverage of open source tools
https://codedx.com/blog/a-look-at-cwe-coverage-across-open-source-and-commercial-static-analysis-tools/ - CAST MRI for Software
https://www.castsoftware.com/products/MRI-for-Software
No comments:
Post a Comment