Characteristics - details
This section contains the following information for each characteristic:
- Definition
- Why is this characteristic important?
Robustness
Definition
Level of degradation suffered by the ML System under consideration when exposed to dynamic or adverse events.
Why is Robustness important?
An ML System should not require disproportionate maintenance in day to day operations and also in face of changes in the environment (such as increased traffic).
Modifiability
Definition
The degree of effectiveness and efficiency with which an ML System can be modified to improve it, correct it or adapt it to changes in environment, and in requirements.
Why is Modifiability important?
A Modifiable ML System delivers value in the long term, when new use cases arise, or when the environment changes and the ML System needs to be adapted. It is important new changes can be implemented with low cost.
Economy
Definition
Level of performance relative to the amount of resources used under stated conditions.
Why is Economy important?
It is important to consider both the cost to impact ratio for an ML System to make sure the system is worth deploying as well as to consider the cost of training, deployment, and inference.
Utility
Definition
The degree to which an ML System provides functions that meet stated and implied needs when used under specified conditions.
Why is Utility important?
An ML systems needs to be useful, in order to bring value to the end user. Utility is about those aspects which make an ML system useful for the end users.
Comprehensibility
Definition
The degree to which users and contributors understand the relevant aspects of an ML system.
Why is Comprehensibility important?
Comprehensibility is about building ML systems that are easy to be understood, used and debugged. Comprehensibility refers both to contributors and users, since some sub-characteristics like Usability refer to the user while others like Debuggability refer to the contributor. A system that is not comprehensible might have limited impact, due to the lack of transparency regarding its usage.
Productionizability
Definition
The ease with which the actions needed for an ML system to operate successfully in production can be performed.
Why is Productionizability important?
An ML system can have value only if it being used in production for a certain use case. Lack of Productionizability means that it is hard to fully exploit the model in production something that might result in limited value.
Responsibility
Definition
The level of trustworthiness of an ML system.
Why is Responsibility important?
The value of an ML system does not only come from its performance on a business metric, but also from its trusthworthiness, e.g. the level of ownership, security, fairness and transparency. A system that is not trustworthy, might end up not being used as it might pose risks that the business cannot afford.