Assignment

• A project proposal: due on 1/21/21, with proposal discussion/feedback in class on 1/22/21.
• A project presentation on 3/5/2021 in class, with the presentation material due on 3/4/2021.
• A final project report: due on 3/8/21.

• It can be either an individual project or a small group project. Unless there is a compelling reason, group size of 3 or more is discouraged due to the expectation that everyone in the team will be performing hands-on testing using multiple testing techniques and submitting a single set of project deliverables.
• Grading: Project grade = report grade; while the proposal is primarily used as a means for you to gather feedback and receive approval from the instructor, and project presentation is primarily for sharing and counted as part of the class participation.
  

The details are given below.

Acceptable project types

1. An application of a couple or more specific formal/systematic testing techniques/models to some programs/products/applications/services you are developing/testing/maintaining/supporting/using/considering. For example, you may choose to:
   • Construct control flow and data flow models and related test cases and perform testing on a module you are developing at your work from a designer/developer's perspective (CFT+DFT/WBT),
   • Construct a finite state machine or input domain partitions to capture the expected behavior of a software or application and test it accordingly from a user's perspective (FSM+PT/BBT),
   • Augment your FSM with quantified usage information into a Markov chain operational profile (OP) to perform usage-based statistical testing from a collective users' perspective (Markov-OP/UBST),
   • Develop/validate/use an operational profile for a large software systems you are working on and collect testing data to quantify its reliability (OP/UBST + SRE).
   and the list goes on...

   It's generally a good idea to consider multiple formal/systematic testing techniques, then select and actually use at least a couple of them in your project to get a hands-on experience of how different techniques work in practical applications. The primary candidate techniques to be considered are the ones we cover in detail in this course, i.e., PT, Musa-OP/UBST, BT, FSM, Markov-OP/UBST, CFT, DFT. Notice that all the techniques mentioned above, except the two UBST ones, can be applied either as a BBT or a WBT technique, resulting in 12 primary formal/systematic testing techniques to select from.
   Important warning: Checklist and other informal/ad-hoc testing are not counted as a formal/systematic testing technique.

   You should pay special attention to the evaluation of your testing/analysis results. Be prepared to answer this type of questions:

   • How do you know if the testing technique works? Consider the applicability, effectiveness, and cost comparisons.
   • What's the basis for comparison (baseline)?
   • What about some other testing techniques that might be appropriate?
   • Is the quality getting better after testing? Can you quantify the trend by the defect metrics or reliability growth?

   Your answer can be based on either the practical evidence (executing several types of testing cases, observing the results, and analyzing the relevant measurement data) or based on logical arguments (suitability of certain testing technique on certain types of products), or both.

   Some analysis of test result, defect, or reliability is expected. Pay particular attention to the collective/overall analysis (e.g., defect/effort/etc. comparison across multiple testing techniques) in addition to individual test result analysis.

2. You may choose to focus on performing various other quality assurance activities and document the results, while performing limited/small-scale (but still needs to be formal and systematic) testing. Possibilities include comprehensive inspection, defect prevention and process improvement, formal verification, fault tolerance, and hazard analysis/resolution for safety-critical systems.. You may also collect inspection/testing/QA records for a product at your work, and construct quality models to analyze the results, to assess the effectiveness of your inspection/testing/QA techniques, or to identify high-defect modules for focused quality improvement actions.

   If you choose this second type of project, I still want you to perform at least some small scale/amount of formal/systematic testing because it is such an integral part of our course (well, we'll devote slightly more than half of our classes to testing alone!).

   For the QA part, make sure you are focusing on quality that can be quantified and/or formally analyzed. For example, some defect analysis, both the root cause analysis type and statistical analysis, or even some classified defect data analysis using schemes such as ODC/AF, and some reliability analysis would be appropriate. However, a process definition/improvement initiative with only a logical argument for its superiority is not suitable for this class.

Where to Find Something to Test or to Perform QA on?

• Well, I mentioned about your work, a product or a system, or a sub-part of it, that you are involved as an architect/designer/developer/tester/user/manager/etc. at your work. However, if you don't feel comfortable taking this as your class project or talking about it (without mentioning product name and raw data) in class, you need to find something else to work on.
  Notice: Your project report must contain testing related technical information, such as test models, test cases, and test results, although you don't have to give detailed product information. As a rule of thumb, if you feel uncomfortable with the material you have to put into this report, find something else to work on.
  Check with your manager about it if you are not sure.

• Many students used their past projects from work, from former college/graduate school classes, from hobby activities (e.g., game programming), or from their role as users of some specific packages/software/web-site/services/etc. Apply the same sensitivity rule as for your current project above.

• You can get it from someone/somewhere else. Open source software and programs are a wonderful source for many student projects in the past as well. In addition, many web-based services/systems/applications can be a good source for your project.

Project proposals

• a project title (be as specific/informative as possible),
• a one-paragraph abstract or an executive summary, at the beginning of the proposal, that summarize your project (not a summary of the software you are going to test),
• a brief introduction: clearly identify the problem that you are going to address,
• brief background information and problems to solve (about 1/2 or 1 page),
• the solution strategy you intend to use (which testing/QA techniques? what kind of analyses? etc., with justification),
• planned activities and artifacts/deliverables to be produced,
• expected results,
• analysis of result to be performed,
• followup actions,
• a rough schedule,

• Please provide information regarding each team member's roles and responsibilities.
• The amount of work proposed for a group project should be appropriate for the group size. As a general rule of thumb, if something can be comfortably done by a single student, it is not suitable as a group project.
• Similarly, more diversity is also expected for a group project.
• You only need to submit one set of deliverables (1 proposal, 1 summary/presentation, and 1 report) for the project, which must follow the same instruction as the individual projects.
• The grade will be assigned to the project, not to the individual members of the team. So, if there are some issues, work them out within your team.

Please keep in mind that by the time you submit your project proposal, we have only covered less than half of the class material, although an overview of the whole course is given at the beginning of the semester. Therefore, you may make certain modifications to the things you propose, but the basic framework should be in your proposal and remain relatively stable.

Once I have reviewed your proposal and provided my feedback, you need to address the issues I raised, in your final project report. However, in most of the cases, you do NOT need to submit a revised proposal. In the rare case that your proposal is marked as "unacceptable" or received 0 grade, I'll explicitly ask you to re-do/re-submit your proposal.

Project presentation

Project report

The format/sequence should be similar to what I listed for the project proposal, but, of course, with more results (what you did, including models and test cases produced, test results, etc., and what you discovered/learned, not just what you proposed to do). Meaningful/descriptive headings/titles for individual numbered sections should be used to organize the report.

If you are not familiar with technical reports or papers, please consult technical/professional journals and conferences proceedings published by ACM, IEEE (particularly its Computer Society), and/or other reputable organizations.

Several common mistakes to avoid:

• It is supposed to be a "report", not a set of "presentation slides". So, limit your use of lists/bullets, and put most of the material/discussions in paragraphs. Similarly, only figures and tables without corresponding discussions do not make a good report.
• Your project report must contain testing related technical information, such as important test models, test cases, test results, and analysis of the results. In addition, you need to describe/discuss this information unless it is clearly self-explanatory.
• On the other hand, you shouldn't include all the graphs, models, test cases, etc. produced for the project in the report main text itself. As I mentioned above, they can be included in the appendix, if you desire, together with other material, such as raw data, customer surveys, selected code/design-doc, etc.
• Again, to avoid the common mistake in the presentation I mentioned earlier of including too much background information but not enough testing/QA-specific technical information, you should focus on the problem-solution-activities-models-results-analysis sequence, preceded by the brief introduction/background and followed by the summary/lessons-learned/etc.