hh.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Locality-Based Test Selection for Autonomous Agents
Halmstad University, School of Information Technology.
Halmstad University, School of Information Technology.
Halmstad University, School of Information Technology. King's College London, London, United Kingdom.ORCID iD: 0000-0002-4869-6794
Quviq Ab, Gothenburg, Sweden.
2022 (English)In: Testing Software and Systems: 33rd IFIP WG 6.1 International Conference on Testing Software Systems, ICTSS 2021, London, UK, November 10-12, 2021 Proceedings / [ed] Clark D., Menendez H., Cavalli A.R., Springer Science+Business Media B.V., 2022, Vol. 13045, p. 73-89Conference paper, Published paper (Refereed)
Abstract [en]

Automated random testing is useful in finding faulty corner cases that are difficult to find by using manually-defined fixed test suites. However, random test inputs can be inefficient in finding faults, particularly in systems where test execution is time- and resource-consuming. Hence, filtering out less-effective test cases by applying domain knowledge constraints can contribute to test effectiveness and efficiency. In this paper, we provide a domain specific language (DSL) for formalising locality-based test selection constraints for autonomous agents. We use this DSL for filtering randomly generated test inputs. To evaluate our approach, we use a simple case study of autonomous agents and evaluate our approach using the QuickCheck tool. The results of our experiments show that using domain knowledge and applying test selection filters significantly reduce the required number of potentially expensive test executions to discover still existing faults. We have also identified the need for applying filters earlier during the test data generation. This observation shows the need to make a more formal connection between the data generation and the DSL-based filtering, which will be addressed in future work. © 2022, IFIP International Federation for Information Processing.

Place, publisher, year, edition, pages
Springer Science+Business Media B.V., 2022. Vol. 13045, p. 73-89
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 13045
Keywords [en]
Autonomous agents, Domain specific languages, Model-based testing, Scenario-based testing, Test input generation, Test selection
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:hh:diva-49111DOI: 10.1007/978-3-031-04673-5_6ISI: 000870696000006Scopus ID: 2-s2.0-85130232932ISBN: 9783031046728 (print)OAI: oai:DiVA.org:hh-49111DiVA, id: diva2:1724119
Conference
33rd IFIP WG 6.1 International Conference on Testing Software Systems, ICTSS 2021, 10 November 2021 through 12 November 2021, , 277709, 2022
Available from: 2023-01-05 Created: 2023-01-05 Last updated: 2024-04-30Bibliographically approved
In thesis
1. Test Automation for Grid-Based Multiagent Autonomous Systems
Open this publication in new window or tab >>Test Automation for Grid-Based Multiagent Autonomous Systems
2024 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Traditional software testing usually comes with manual definitions of test cases. This manual process can be time-consuming, tedious, and incomplete in covering important but elusive corner cases that are hardly identifiable. Automatic generation of random test cases emerges as a strategy to mitigate the challenges associated with the manual test case design. However, the effectiveness of random test cases in fault detection may be limited, leading to increased testing costs, particularly in systems where test execution demands substantial resources and time. Leveraging the domain knowledge of test experts can guide the automatic random generation of test cases to more effective zones. In this thesis, we target quality assurance of multiagent autonomous systems and aim to automate test generation for them by applying the domain knowledge of test experts.

To formalize the specification of the domain expert's knowledge, we introduce a small Domain Specific Language (DSL) for formal specification of particular locality-based constraints for grid-based multiagent systems. We initially employ this DSL for filtering randomly generated test inputs. Then, we evaluate the effectiveness of the generated test cases through an experiment on a case study of autonomous agents. Applying statistical analysis on the experiment results demonstrates that utilizing the domain knowledge to specify test selection criteria for filtering randomly generated test cases significantly reduces the number of potentially costly test executions to identify the persisting faults. 

Domain knowledge of experts can also be utilized to directly generate test inputs with constraint solvers. We conduct a comprehensive study to compare the performance of filtering random cases and constraint-solving approaches in generating selective test cases across various test scenario parameters. The examination of these parameters provides criteria for determining the suitability of random data filtering versus constraint solving, considering the varying size and complexity of the test input generation constraint. To conduct our experiments, we use QuickCheck tool for random test data generation with filtering, and we employ Z3 for constraint solving. The findings, supported by observations and statistical analysis, reveal that test scenario parameters impact the performance of filtering and constraint-solving approaches differently. Specifically, the results indicate complementary strengths between the two approaches: random generation and filtering approach excels for the systems with a large number of agents and long agent paths but shows degradation in larger grid sizes and stricter constraints. Conversely, constraint solving approach demonstrates robust performance for large grid sizes and strict constraints but experiences degradation with increased agent numbers and longer paths.

Our initially proposed DSL is limited in its features and is only capable of specifying particular locality-based constraints. To be able to specify more elaborate test scenarios, we extend that DSL based on a more intricate model of autonomous agents and their environment. Using the extended DSL, we can specify test oracles and test scenarios for a dynamic grid environment and agents having several attributes. To assess the extended DSL's utility, we design a questionnaire to gather opinions from several experts and also run an experiment to compare the efficiency of the extended DSL with the initially proposed one. The questionnaire results indicate that the extended DSL was successful in specifying several scenarios that the experts found more useful than the scenarios specified by the initial DSL. Moreover, the experimental results demonstrate that testing with the extended DSL can significantly reduce the number of test executions to detect system faults, leading to a more efficient testing process.

Place, publisher, year, edition, pages
Halmstad: Halmstad University Press, 2024. p. 21
Series
Halmstad University Dissertations ; 116
Keywords
Test Input Generation, Domain Specific Languages, Test Selection, Autonomous Agents, Scenario-based Testing
National Category
Software Engineering Computer Sciences
Identifiers
urn:nbn:se:hh:diva-53268 (URN)978-91-89587-49-6 (ISBN)978-91-89587-48-9 (ISBN)
Presentation
2024-05-24, Wigforss salen, hus J, Kristian IV:s väg 3, Halmstad, 13:00 (English)
Opponent
Supervisors
Projects
Safety of Connected Intelligent Vehicles in Smart Cities – SafeSmart
Funder
Knowledge Foundation
Available from: 2024-04-30 Created: 2024-04-30 Last updated: 2024-05-03Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Mostowski, WojciechMousavi, Mohammad Reza

Search in DiVA

By author/editor
Entekhabi, SinaMostowski, WojciechMousavi, Mohammad Reza
By organisation
School of Information Technology
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 33 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf