Grading

Evaluation criterion and evaluation scale 

Evaluation criterion description

Comments and examples 

Correctness
[-1/ 0] 

The risk calculations, conclusions, and other syntheses are made correctly.  

The fusion of parameters should follow the described method or otherwise make sense and be motivated. E.g. it doesn’t make sense to sum the attacker capability with a vulnerability score. And the calculations must match the developed models. 

Consistency  
[-1/ 0-2]

The threat models follow the metamodel (Yacraf, normally). The models are consistent within each sub model (within a phase) and between one another (across the phases). The same phenomenon is modelled in the same way or transparently transformed between models. Are the different models interconnected to form a single underlying unambiguous model. Risk calculations are done according to the same framework for all scenarios and models. Terminology is used consistently (and correctly).

E.g.
- Are multiplicities from the modeling language followed and are phenomena modeled according to the logic of the language? 
- "Threat," "risk," and "vulnerability" should mean different things (unless something else is explicitly stated) and the meaning of the respective words should be the same throughout the report.
- Are the same assets modelled in the various DFDs to form a more complete architecture?
- Are the vulnerabilities discussed in relation to the assets in the system architecture
Do the attack vectors/trees relate to the identified vulnerabilities and the assets.,
- Is it possible to follow/track/relate all model elements to one another.

The consistency should be demonstrated (or rather the report should not leave places where the consistency can be doubted or questioned). 

Coverage 
[-1/ 0-1]

Given a decided scope the models and analysis should cover this scope in a reasonably complete way. 

E.g. for attack models - apparent critical attack vectors should not be missing, for system models - key components should not be missing, for business goals - obvious losses should not be missing. 

The Yacraf metamodel should also be broadly/fully covered in the instance models. 

Variance 
[-1/ 0-2]

The models cover a variety of phenomena.  

Not all models/analyses should follow and repeat the same underlying logic. E.g. a model containing three web servers with very similar vulnerabilities. (Of course the system may surely have three similar servers but with respect to this particular criteria they do not "add value".) One should demonstrate ability to analyze a wide variety of security topics. Different types of systems with different types of attack vectors and vulnerabilities.  

Technical Detail
[-1/ 0-3]

The models should be on such a depth so that they are not trivial or superficial. Also, it should be clear what the technical challenges / implementations / weaknesses / attacks are.

This criterion is specifically addressing the the Yacraf phases 2 and 4 with system and attack models.  

A (too) high-level description of an attack vector would e.g. be "DoS web server <- deploy botnet <- buy botnet", where a corresponding high-level remedy would be: "buy load balancer".

For the system models more detail e.g. mean configurations of the inner logic of a component and its underlying platforms. 

Note! Adding technical detail just for the sake of it is not the purpose of this criterion. The details much also be well balanced (see separate criterion) so that technical detail of a system is relevant in relation to some adressed attack vector. 

Balance
[-1/ 0-1]

Balance the level of detail to avoid that one part/aspect of the models and analysis gets a lot of attention while others get a little attention.  

Balance should be maintained between different Yacraf phases. It also applies to the different parts of the model so that one part of the system or business is analysed in great detail while another part is very coarsely described.

Realism and motivation 
[-1/ 0-3]

The model input data should be realistic, but not real. Your model and analysis should be of such characteristic that it could have been describing something that could have been real.

Some things are straight forward and not questionable while for some assumptions, interpretations, motivations, and external references are needed to support the claims of the models.

In principle there are two ways of motivating non-questionable claims, either (trustworthy) references are put forward as support, or you put forward your own argumentation that is convincing the reader about the realism. This can relate to anything from costs of loss events to common system architecture solutions and technology stacks, to threat profiles and attack vectors. 

Strong references, and use of, external sources should be used. E.g. analysis according to some certain MITRE ATT&CK techniques, or some vulnerability class, and the use of threat emulation profiles. 

For the threat analysis there are many parameters that is qualitatively assessed (e.g. prob. of action). For these assessments some form of reasoning / motivation is needed.   

Motivating why you are not diving deeper into something is a good thing. E.g. "we consider TLS 1.3 secure, because ... "

Assurance 
[-1/ 0-2]

The confidence of the final  recommendations in the report should be high.

This relates to the amount of different mitigation scenarios that are analysed, but also how they are compared against each other in relation to how viable or costly they are to implement. 

This criterion also relates to the remaining uncertainty after the realism has been motivated. This should be explicitly described and reflected upon. 

It also includes that the scenarios are well balanced in comparison to each other with respect to detail, coverage, realism, etc. It should also be clear thet it is the analysis that is the motivating the recommendations.  

Readability
[-1/ 0]

The report must be well-structured, understandable and easy to follow.

When it comes to the formatting and formalities the report must for instance be free of spelling errors, inconsistent or incomprehensible sentences, and grammatical errors. It must have reasonable size of paragraphs and sentences. It must use numbered headings and it must be easy to follow all internal references in the report. It must use figure/table text and reference figures/tables in text. (E.g. figure text description: “Figure 1. A business model”. In text: ”In Figure 1 the business model is described” .)