Threats to Validity

1. Conclusion validity threats

Conclusion validity threats are related to factors that can affect conclusions drawn from experimental results.

To deal with such a threat, we ran the experiment 100 runs (Run as many experiments as possible to remove randomness).

2. External validity threats

External validity threats concern the generalization of the results.

To alleviate this threat, we try to make our subjects' systems realistic by selecting real-world cyber-physical programs. Besides, we make the subjects diverse by requesting them to cover different functionalities (e.g., automated area exploration, planned flying, and smart obstacle avoidance), and to run on different platforms.

3. Construct validity threats

Construct validity threats refer to the degree to which the experiment setting (e.g., metrics) reflects the construct under study (i.e., the quality, the cost-effectiveness, and the performance of the proposed approach). 

or Construct validity is concerned with how well the metrics used in the experiment reflect the construct.

To reduce the threats, we carefully selected and defined the metrics focusing on our overall objective of study under testing.

4. Internal validity threat

Internal validity threat refers to the influence that affects the causal relationship between the treatment and outcome.

To reduce the threats, different experiment settings might favor one algorithm over the other. However, more experiments with other settings in terms of test execution time (or other metrics) are required to further strengthen the current conclusion.