CS 7314/5314 (Spring 2024): Course Project
Assignment
Your course project is an integral and the most important part
of your CS 7314/5314,
an opportunity for you to apply the knowledge learned in your class,
particularly software testing (and possibly other QA/quality modeling)
techniques,
in a realistic setting,
and to report related experience, findings, and lessons learned.
It will consist of three parts or stages:
-
A project proposal:
due on 2/8/24.
-
A project progress report
due on 3/21/24;
or an optional (formal) project presentation
to be scheduled for the last 4-5 weeks of classes.
-
A final project report:
due on 4/25/24.
Notice:
This part is slightly different for graduate students
and undergraduate students. See below.
Some general information:
-
It can be either an individual project or a small group project.
Unless there is a compelling reason,
group size of 3 or more is discouraged due to the expectation that
everyone in the team will be performing hands-on testing
using multiple testing techniques
and submitting a single set of project deliverables.
-
There is also slightly different requirements for graduate
and undergraduate students in the result analysis part of
your project -- see details below.
-
The proposal is primarily used as a means
for you to gather feedback and receive approval from the instructor.
Observation:
I have yet to see an excellent project
without getting the proposal feedback first.
-
I'll provide some feedback to your progress report in writing
or verbally for those of you giving presentations.
-
Important:
Regardless of the type of project you select,
you must do something (testing, QA, quality modeling)
specifically for this class,
not something you did earlier,
and submit a specifically prepared set of project deliverables.
Similarly,
a simple documentation of on-going testing/QA at your work
or at other places is not acceptable.
-
Caution:
Submission of someone else's work/report/paper
will automatically get a 0 grade for the project assignment.
And, you may be referred to SMU honor council for the violation of
SMU honor code.
The details are given below.
Acceptable testing projects
Your course project will focus on testing,
which may also include
on other QA techniques or quality modeling (see next section below).
For an individual project,
it should include an application of at least 2 specific formal/systematic
testing techniques/models
to some programs/products/applications/services
you are developing/testing/maintaining/supporting/using/considering.
For example, you may choose to:
- Construct control flow and data flow models
for a module you are developing at your work
from a designer/developer's perspective;
generate related test cases from these models;
and perform testing accordingly
(CFT+DFT/WBT),
- Construct a finite state machine or input domain partitions
to capture the expected behavior of a software or application
from a user's perspective,
and test it accordingly (FSM/BBT or PT/BBT),
- Augment your FSM with quantified usage information into a Markov
chain operational profile (OP) to perform usage-based statistical
testing from a collective users' perspective (Markov-OP/UBST),
- Develop/validate/use an operational profile
for a large software systems you are
working on and collect testing data to quantify its reliability
(OP/UBST + SRE).
and the list goes on...
It is generally a good idea to consider multiple formal/systematic
testing techniques,
preferably from different categories
(e.g., both flat models and FSM-based models)
and perspectives (e.g., BBT, WBT, and UBST),
and then select and actually
use at least 2 of them in your project to get a hands-on experience
of how different techniques work in practical applications.
Here is the rule regarding what can be counted as a formal/systematic
testing technique for your course project:
-
Yes:
The primary candidate techniques to be considered are the ones we cover
in detail in this course,
i.e., PT, Musa-OP/UBST, BT, FSM, Markov-OP/UBST, CFT, DFT.
Notice that all the techniques mentioned above, except the two UBST ones,
can be applied either as a BBT or a WBT technique,
resulting in 12 primary formal/systematic testing techniques to select from.
-
No:
Checklist and other informal/ad-hoc testing
are not counted as a formal/systematic testing technique.
If you start with a checklist, it must be formalized
into PT/FSM/etc.
to be counted as a formal/systematic testing technique.
-
No:
Don't confuse a testing technique with a testing subphase
(e.g., unit testing, component testing, stress testing, etc.),
or testing of a specific type of products or for a specific purpose
(e.g., Web testing, usability testing, security testing, etc.).
If you are performing some testing in this category,
you still need to select your specific testing techniques.
-
Maybe:
For testing technique not falling into the above categories,
please check with the instructor.
Pay special attention to the evaluation of your testing/analysis results,
especially the collective/overall analysis
(e.g., defect/effort/etc. comparison across multiple testing techniques),
in addition to individual test result analysis.
Be prepared to answer the following questions:
- How do you know if the testing technique worked/works?
Consider the applicability, effectiveness, and cost comparisons.
- What's the basis for comparison (baseline)?
- What about some other testing techniques that might be appropriate?
- Is the quality getting better after testing?
Can you quantify the trend by the defect metrics or reliability growth?
Your answer can be based on either the practical evidence
(executing several types of testing cases,
observing the results, and analyzing the relevant measurement data)
or based on logical arguments
(suitability of certain testing technique on certain types of products),
or both.
- Graduate students are expected to study relevant material from Part IV of
our textbook to perform the analyses above.
Some test result, defect, or reliability analyses are expected.
- Undergraduate students can choose to perform only a high-level analysis
to answer the above questions without going into detailed data analysis.
QA/SQE beyond testing in your project
Your project may also include
other QA techniques or quality modeling (part of the SQE process),
as described below:
-
You may choose to perform various other quality assurance
activities and document the results,
as a substitute for one of the formal/systematic testing technique
required.
Possibilities include comprehensive inspection,
defect prevention and process improvement,
formal verification, fault tolerance,
and hazard analysis/resolution for safety-critical systems.
-
You may also collect inspection/testing/QA records
for a product at your work,
and construct quality models to analyze the results,
to assess the effectiveness of your inspection/testing/QA techniques,
or to identify high-defect modules for focused
quality improvement actions.
Make sure you are focusing on quality that can be quantified and/or
formally analyzed.
For example, some defect analysis, both the root cause analysis type
and statistical analysis, or even some classified defect data analysis
using schemes such as ODC/AF,
and some reliability analysis would be appropriate.
However,
a process definition/improvement initiative with only a
logical argument for its superiority is not suitable
for this course project.
Where to Find Something to Test or to Perform QA on?
-
Well, I mentioned about your work,
a product or a system, or a sub-part of it,
that you are involved as an
architect/designer/developer/tester/user/manager/etc.
at your work.
However, if you don't feel comfortable taking this as your class
project or talking about it (without mentioning product name and
raw data) in class, you need to find something else to work on.
Notice:
Your project report must contain testing related technical information,
such as test models, test cases, and test results,
although you don't have to give detailed product information.
As a rule of thumb, if you feel uncomfortable with the material
you have to put into this report, find something else to work on.
Check with your manager about it if you are not sure.
-
Because of the ubiquitous nature of software and systems today,
everyone is a user of some software/system/service/etc.,
which can be tested from a user's perspective.
Similarly,
many web-based services/systems/applications can be a good source
for your project.
Of course, if you have access to the source code,
from someone/somewhere else,
you'll be able to perform WBT as well.
Open source software and programs are a wonderful source for many
student projects in the past as well.
-
Many students used their past projects from work,
from former college/graduate school classes,
from hobby activities (e.g., game programming),
or from their role as users of some specific
packages/software/web-site/services/etc.
Apply the same sensitivity rule as for your current project above.
Project proposal
Your project proposal should be around 3-4 double spaced pages
in length, and should include the following information:
- a project title (be as specific/informative as possible),
- a one-paragraph abstract or an executive summary,
at the beginning of the proposal,
that summarize your project
(not a summary of the software you are going to test),
- a brief introduction,
- brief background information and problems to solve (about 1/2 or 1 page),
clearly identify the problem that you are going to address,
- the solution strategy you intend to use
(starting with the overall strategy, and
then individual testing/QA techniques to be used
kind of analyses to be performed, etc.,
with justification),
- planned activities and artifacts/deliverables to be produced,
- expected results,
- analysis of result to be performed,
- followup actions,
- a rough schedule,
- indicate whether you'll be making a project presentation,
and, if possible, your preferred presentation date.
In case of a group project, please also pay attention to the following:
-
Please provide information regarding each team member's
roles and responsibilities.
-
The amount of work proposed for a group project should be appropriate
for the group size. As a general rule of thumb, if something can be
comfortably done by a single student, it is not suitable as a group
project.
-
Similarly, more diversity is also expected for a group project.
-
If you choose to perform multiple other QA/SQE activities (see above),
you may substitute for two of the formal/systematic
testing techniques required for such a team project.
-
You only need to submit one set of deliverables
(1 proposal, 1 progress-report/presentation, and 1 report)
for the project,
which must follow the same instruction as the individual projects.
-
If your team consists of both undergraduate and graduate students,
the rules for graduate students will apply.
-
Similarly, if your team consists of both on-campus and distance students,
the team will be treated as an on-campus team.
-
The grade will be assigned to the project,
not to the individual members of the team.
So, if there are some issues, work them out within your team.
Please keep in mind that by the time you submit your project proposal,
we have only covered less than half of the class material,
although an overview of the whole course is given at the beginning of
the semester.
Therefore,
you may make certain modifications to the things you propose,
but the basic framework should be in your proposal and remain
relatively stable.
Once I have reviewed your proposal and provided my feedback,
you need to address the issues I raised,
in your final project report.
However, in most cases,
you do not need to submit a revised proposal.
In the rare case that your proposal is marked as "unacceptable"
or received 0 grade,
I'll explicitly ask you to re-do/re-submit your proposal
via e-mail.
Progress report
All the students are required to submit a progress report,
if you are not doing a full presentation (see below) in class.
Your progress report should focus on project progress you made so far,
i.e., the main activities and results from your project
after the proposal submission/review/feedback cycle.
Here are some specifics about the progress report:
-
Your progress report should be around 3-4 double spaced pages
in length,
or about 6-8 slides in presentation slide format.
-
A brief summary of what you proposed earlier should be included,
1/2 to 1 page.
-
Focus: progress so far, post-proposal stage,
including data collected for model construction,
testing models constructed,
test cases constructed and executed,
testing results,
result analysis, etc.
-
What remain to be done (progress vs plan/proposal),
and when.
I'll give you written feedback for all the progress reports submitted,
which should be incorporated into your final project report.
Project presentation
You may choose to do a (full) project presentation.
In that case, you don't need to submit the progress report.
Each presentation should last about 10-15 minutes,
with an appropriate number of slides.
You need to highlight the problem/solution-strategy/results/analysis
for us to get the basic picture,
but not necessarily all the details,
which would require much more than 15 minutes.
One common mistake in the past is too much background information
but not enough testing/QA-specific technical information.
Also, avoid detailed description of the test cases and activities performed.
For distance students interested in making a presentation:
I'd encourage you to make an effort to come to the campus to give your
presentation to provide the opportunity for interactive discussions.
However, if you have difficulty with it,
pre-recorded presentation can also be acceptable.
I'll give you verbal feedback during your presentation,
which should be incorporated into your final project report.
Project report
The project report should be treated as a term paper of your own,
around 15 double-spaced pages in length,
but no longer than 20 pages, for graduate students.
The report should clearly and comprehensively states the background,
problem, strategy, activities, models, test cases, results, result analysis,
lessons learned, followup actions, and a high level summary
(and an abstract at the beginning).
Additional material, such as graphs, models, test cases, etc. produced,
information sources and raw data,
customer surveys, etc., can be included in the appendix and clearly
marked as such (so it will not be counted towards your 20 page quota).
However, graphs/models/tables/etc. central to the report should be
included int the report itself, not in the appendix.
For undergraduate students, you report should be similar, except the
analysis part (see information in the "Proposal" section above).
As a result, your report is expected to be around 10-12 pages.
But, if you choose to, you can follow the instruction for graduate students
as well.
The format/sequence should be similar to what I listed for
the project proposal,
but, of course, with more results
(what you did,
including models and test cases produced,
test results, result analysis, etc., and what you discovered/learned,
not just what you proposed to do).
Meaningful/descriptive headings/titles for individual numbered
sections should be used to organize the report.
If you are not familiar with technical reports or papers,
please consult technical/professional journals and conferences
proceedings published by ACM, IEEE (particularly its Computer Society),
and/or other reputable organizations.
Several common mistakes to avoid:
-
It is supposed to be a "report",
not a set of "presentation slides".
So, limit your use of lists/bullets, and put most of the material/discussions
in paragraphs.
Similarly,
only figures and tables without corresponding discussions
do not make a good report.
-
Your project report must contain testing related technical
information, such as important test models,
summary of test cases and results
(and discussion of some important ones, if appropriate),
and analysis of the results.
Use tables and/or figures if appropriate.
In addition,
you need to describe/discuss this information
(particularly the tables/figures included).
-
On the other hand,
you shouldn't include all the graphs, models, test cases, etc. produced
for the project in the report main text itself.
As I mentioned above, they can be included in the appendix, if you desire,
together with other material, such as raw data,
customer surveys, selected code/design-doc, etc.
Notice: the material included in the appendix will not count towards
the page limit.
-
Again, to avoid the
common mistake in the presentation I mentioned earlier
of including too much background information
but not enough testing/QA-specific technical information,
you should focus on the problem-solution-activities-models-results-analysis
sequence, preceded by the brief introduction/background and followed by
the summary/lessons-learned/etc.
Prepared by Jeff Tian
(tian@smu.edu).
Initially posted: Jan. 31, 2024.
Last update: Feb. 2, 2024.