Software Testing and Quality Assurance

Contents Description

One important feature of this course is the user-centered perspective of software quality, testing, and QA. User requirements for new systems and usage models of existing systems or alternative systems will be the starting point and focus of our course. Quality metrics meaningful to the users will be used throughout the QA and SQE (Software Quality Engineering) process.

Another important feature of this course is the focus on the "prioritized" (or "risk-based") approach to testing and quality assurance. For any realistic system today, exhaustive testing and similar quality assurance approaches would be infeasible because of the increasing size and complexity. We will focus on the "kay" areas/goals/components/etc., related to importance/criticality (externally defined priority), likely defects (defect-based), usage frequencies (usage-based), etc., for our testing and QA effort.

We will devote slightly more than half of the class time to topics related to software testing, or executing the software in order to observe its behavior to ensure that it conforms to our expectations or to identify behavior deviations and possible underlying problems for correction. We will emphasis formal/systematic testing techniques and their applications, including:

• black-box or functional testing, with a focus on covering (or thoroughly testing) external functions or services required or expected by target customers and users;
• clear/transparent/white-box or structural testing; with a focus on covering (or verifying/testing) implementation details, including internal components and structures;
• usage-based statistical testing (UBST), with a focus on adequately testing operations or services important to and commonly used by target customers and users to ensure overall product or system reliability (a quality view most meaningful to customers and users) after product delivery or system deployment.

• Testing techniques based on "flat" structures, such as lists, (top-down) partitions and (bottom-up) equivalent classes, include t1) basic partition testing (PT), t2) input domain boundary testing (BT), and t3) Musa's operational profiles (Musa OP) or usage model for UBST.
• Testing techniques based on finite-state machines (FSM) and related variations include t4) basic FSM testing to, t5) control flow testing (CFT), t6) data dependency analysis and data flow testing (DFT), and t7) Markov operational profiles (Markov OP) for UBST.
• Notice that t3 and t7 above are usage-based, while the others can be either BBT or WBT in perspective, resulting in a total number of 12 variations of formal/systematic testing techniques.
  (Note: These testing techniques will be the candidates to be selected for implementation in your course project. -- More details later.)

Specific testing techniques to be covered: Besides the formal/systematic ones mentioned above (t1-t7: PT, BT, Musa-OP/UBST, FSM, CFT, DFT, Markov-OP/UBST), we will also briefly cover various forms of informal or ad-hoc testing, checklists, fault injection and mutation testing. Specialized testing techniques for specific purposes and/or applicable to different application domains, such as usability testing, Web testing, Cloud (and API) testing, embedded system testing, performance and stress testing, etc. will also be briefly covered.

Test activities, management, automation, and related issues, such as team organization, testing process, people's roles and responsibilities, user involvement (user-centered testing/QA), test automation tools, test integration, etc., will be briefly discussed. Applicability and effectiveness of specific testing techniques in different sub-phases and for different purposes will also be briefly discussed.

The rest of the class time, beyond various testing-related topics mentioned above, will be devoted to the following topics:

• General concepts about quality, quality assurance (QA), and software quality engineering (SQE). Concepts related to quality will be covered, including reliability, safety, security, usability, dependability, maintainability, availability, etc., primarily from a user's perspective. QA activities and SQE process will also be discussed, in the context of overall software development and maintenance processes, with customer involvement.
• Other QA alternatives beyond testing, including, activities and related techniques for defect prevention and process improvement, inspection/review/walkthrough, formal verification (and its prerequisite, formal specification, briefly), fault tolerance and safety assurance. A good understanding of each alternative's relative strengths and weaknesses will help us construct and carry out a comprehensive QA strategy that more often than not will include and integrate several such alternatives.
• Quality assessment and improvement, including, overall strategy for quantifiable quality improvement, measurement and feedback mechanisms, quality models and measurements, defect classification and analysis, risk identification and analysis, and software reliability engineering. These activities will help us monitor, assess, improve the overall SQE process to achieve our quality and reliability goals effectively and efficiently.