CDN (Diego) Damasceno

Active automata learning is a technique for dynamically learning finite state machine models of black-box systems. Conformance testing is a well-known bottleneck during learning. While multiple conformance testing techniques (CTTs) for Finite State Machines have been proposed, there is a lack of empirical studies that assess the effects of these CTTs \emphduring learning. In this work, we compare the performance of eight different CTTs (W, Wp, HSI, H-ADS, H, SPY, SPY-H, I-ADS) while learning 46 models from different communication protocols. Moreover, we propose \mathttAPFD_L as a metric for characterizing the efficiency of automata learning experiments in terms of fault detection capacity. This metric allows identifying CTTs with a lower total cost regarding the number of symbols and resets and a higher rate of state discovery during learning. Our results indicate that while the total cost entailed by CTTs in learning tends to be negligible, we found a significant difference in fault detection rate in learning. Nevertheless, the differences in fault detection rates become negligible when CTTs are applied in randomized mode. These findings reveal the positive role that randomness can have in improving learning efficiency, despite compromising test completeness.

Behavioral models enable the analysis of the functionality of software product lines (SPL), e.g., model checking and model-based testing. Model learning aims to construct behavioral models. Due to the commonalities among the products of an SPL, it is possible to reuse the previously-learned models during the model learning process. In this paper, an adaptive approach, called PL*, for learning the product models of an SPL is presented based on the well-known L* algorithm. In this method, after learning each product, the sequences in the final observation table are stored in a repository which is used to initialize the observation table of the remaining products. The proposed algorithm is evaluated on two open-source SPLs and the learning cost is measured in terms of the number of rounds, resets, and input symbols. The results show that for complex SPLs, the total learning cost of PL* is significantly lower than that of the non-adaptive method in terms of all three metrics. Furthermore, it is observed that the order of learning products affects the efficiency of PL*. We introduce a heuristic to determine an ordering which reduces the total cost of adaptive learning.

Sharing research artifacts is known to help people to build upon existing knowledge, adopt novel contributions in practice, and increase the chances of papers receiving attention. In Model-Driven Engineering (MDE), openly providing research artifacts plays a key role, even more so as the community targets a broader use of AI techniques, which can only become feasible if large open datasets and confidence measures for their quality are available. However, the current lack of common discipline-specific guidelines for research data sharing opens the opportunity for misunderstandings about the true potential of research artifacts and subjective expectations regarding artifact quality. To address this issue, we introduce a set of guidelines for artifact sharing specifically tailored to MDE research. To design this guidelines set, we systematically analyzed general-purpose artifact sharing practices of major computer science venues and tailored them to the MDE domain. Subsequently, we conducted an online survey with 90 researchers and practitioners with expertise in MDE. We investigated our participants’ experiences in developing and sharing artifacts in MDE research and the challenges encountered while doing so. We then asked them to prioritize each of our guidelines as essential, desirable, or unnecessary. Finally, we asked them to evaluate our guidelines with respect to clarity, completeness, and relevance. In each of these dimensions, our guidelines were assessed positively by more than 92% of the participants. To foster the reproducibility and reusability of our results, we make the full set of generated artifacts available in an open repository at https://mdeartifacts.github.io/.

Family-based behavioral analysis operates on a single specification artifact, referred to as family model, annotated with feature constraints to express behavioral variability in terms of conditional states and transitions. Family-based behavioral modeling paves the way for efficient model-based analysis of software product lines. Family-based behavioral model learning incorporates feature model analysis and model learning principles to efficiently unify product models into a family model and integrate the behavior of various products into a behavioral family model. Albeit reasonably effective, the exhaustive analysis of product lines is often infeasible due to the potentially exponential number of valid configurations. In this paper, we first present a family-based behavioral model learning techniques, called FFSMDiff. Subsequently, we report on our experience on learning family models by employing product sampling. Using 105 products of six product lines expressed in terms of Mealy machines, we evaluate the precision of family models learned from products selected from different settings of the T-wise product sampling criterion. We show that product sampling can lead to models as precise as those learned by exhaustive analysis and hence, reduce the costs for family model learning.

Access control mechanisms demand rigorous software testing approaches, otherwise they can end up with security flaws. Finite state machines (FSM) have been used for testing Role-Based Access Control (RBAC) mechanisms and complete, but significantly large, test suites can be obtained. Experimental studies have shown that recent FSM testing methods can reduce the overall test suite length for random FSMs. However, since the similarity between random FSMs and these specifying RBAC mechanisms is unclear, these outcomes cannot be necessarily generalized to RBAC. In this paper, we compare the characteristics and effectiveness of test suites generated by traditional and recent FSM testing methods for RBAC policies specified as FSM models. The methods W, HSI and SPY were applied on RBAC policies specified as FSMs and the test suites obtained were evaluated considering test characteristics (number of resets, average test case length, and test suite length) and effectiveness on the RBAC fault domain. Our results corroborate some outcomes of previous investigations in which test suites presented different characteristics. On average, the SPY method generated test suites with 32% less resets, average test case length 78% greater than W and HSI, and overall length 46% lower. There were no differences among FSM testing methods for RBAC regarding effectiveness. However, the SPY method significantly reduced the overall test suite length and the number of resets.