Mobile Applications for Sensing and Learning CSE5323 & CSE7323

Downloads

Featured Final Project: Explore@SMU This was one of my favorite projects in the 2015 offering of the course (among many of them!). A description of the project can be downloaded here. Be sure to also check out this link to the final project presentation and the summary video for the app here.

Templates for each assignment can be found on blackboard with slides and videos. Please see blackboard for class content.
    Outside Links


    Grading Rubric

    In all assignments specific deliverables are asked and should be completed to the best of your team's ability. Each deliverable will be worth a certain percentage of the grade and you will be graded in terms of how efficient your implementation is, how well you use proper coding styles and interface guidelines (including proper use of model-view-controller), and how comprehensible each element is integrated. Make it memory efficient and computationally efficient as possible and use hardwrae acceleration. Comment code so that it is readable and immediately understandable. The sum total of the these deliverables will be 90% of the points possible for each lab assignment. Teams that complete all the project deliverables satisfactorily should expect a grade of 90%.

    The remaining 10% of the points are reserved for truly exceptional work and work that is above and beyond in one or more elements of the application, incorporating elements not discussed in class and having superior performance. These are applications that, while perhaps are not ready to ship on the iTunes app store, are production quality and require minor changes to be ready for consumer use.  

    Assignment One - Sink or Swim iOS Interface Building

    You have free reign to create an application that manages some type of mutable information. That means that the data displayed by the app is expected to change. The data you load and display can come from anywhere and you can do whatever you want with it. For instance, you might display images from online somewhere, you might display stock exchange information, or you might display a information from twitter--or movies, or books, or amazon. It can be whatever information you want to display (really, have fun with it), as long as you use the interface elements as described below. You will need to get creative in order to incorporate ALL the design elements below. That is okay, some parts of the app will just be weird (but not all the app).

    Create an iOS application in XCode that uses the storyboard and a TableViewController to load different views based on the data your are loading. The TableViewController must implement three different types of cells and load them dynamically (i.e., you cannot use a static table). View navigation can be hierarchical in any way you want--as long as it makes sense for the interface and the data you are displaying. When loading a new view controller your main view controller should hand off information to the controller that is getting created. The application should make use of the following design elements in one or more of the views:
    • Automatic Layout
    • Buttons, Sliders, and Labels
    • Stepper and Switch
    • Picker (you must implement picker delegate)
    • Segmented Control
    • Timer (which should repeat and somehow update the UIView)
    • ScrollView (with scrollable, zoomable content)
    • Image View
    • Navigation Controller
    • Collection View Controller
    • Table View Controller with dynamic prototype cells
    • (Extra credit) Implement a modal view and handle properly using delegation
    The design should work in both portrait and landscape mode. I should not be able to crash your app. Test your app running on the device, not the emulator to ensure it runs in all scenarios. Your design must strictly adhere to Model View Controller programming practices. Use lazy instantiation when possible.

    Turn in the source code for your app in zipped format via blackboard. Use proper coding techniques and naming conventions for objective C and/or swift. Use whichever programming language you are most comfortable with. Include your team member names and team name in the comments of the "main.m" file as well as in blackboard upload text. Upload as "teamNameAssignmentOne.zip".

    Remember to also fill in the team member contribution form and back up all the contributions that you assign.

    Assignment Two - Audio Filtering, the FFT, and Doppler Shifts

    Module A
    Create an iOS application using the example template that:
    • Reads from the microphone
    • Takes an FFT of the incoming audio stream
    • Displays the frequency of the two loudest tones within 6Hz accuracy (+-3Hz)
      • Please have a way to "lock in" the last frequencies detected on the display
    • Is able to distinguish tones at least 50Hz apart, lasting for 200ms or more
    • Extra credit: recognize two tones played on a piano (down to one half step apart) and report them by letter (i.e., A4, A#4). Must work at note A2 and above. Note: this is harder than just identifying two perfect sine waves!!
    Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email). The sound source must be external to the phone (i.e., laptop, instrument, another phone, etc.).

    Module B
    Create an iOS application using the example template that:
    • Reads from the microphone
    • Plays a settable (via a slider or setter control) inaudible tone to the speakers (15-20kHz)
    • Displays the magnitude of the FFT of the microphone data in decibels
    • Is able to distinguish when the user is {not gesturing, gestures toward, or gesturing away} from the microphone using Doppler shifts in the frequency
    Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email).

    Turn in the source code for your app in zipped format via blackboard. Use proper coding techniques and naming conventions for  swift, objective C, and objective C++. Include your team member names and team name in the comments of the "main.m" files as well as in blackboard upload text. Upload as "teamNameAssignmentTwo.zip".

    Remember to also fill in the team member contribution form and back up all the contributions that you assign.

    Grading Rubric
    • [5 points] Proper Interface Design (navigation and auto layout)
    • [25 points] Algorithms Design (efficiency, argmax finding, proper sampling, and ring buffer use, etc.)
    • [30 points] Module A (frequency displayed, 6Hz accuracy, 200 ms duration, 50 Hz difference)
    • [30 points] Module B (tone is settable, FFT displayed properly, Accurate gesture detection across all settable tones)
    • [10 points] Exceptional (free reign to make updates: perhaps using the gestures as a form of control in the app, or implementing more than just towards/away detection, etc.)
    • [5 points] Extra Credit

    Assignment Three - Core Motion and SceneKit

    Module A
    Create an iOS application that:
    • Displays the number of steps a user has walked today and yesterday
    • Displays a realtime count of the number of steps a users has taken today
    • Displays the number of steps until the user reaches a (user settable) daily goal
    • Displays the current activity of the user: {still, walking, running, cycling, driving}
    Module B
    Create an additional part of the app that, whenever the user meets their step goal for the previous day, allows the playing of a simple game (it can be very simple). The game must:
    • Uses {acceleration, gyro, magnetometer, OR fused motion} to control some part of the physics of a SceneKit game
    • Uses two or more SceneKit objects with dynamic physics
    • An idea for exceptional work: use the steps of a user as some type of "currency" in the game to incentivize movement during the day
    The application should make use of the M7/M8 co-processor whenever possible. Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email).

    Turn in the source code for your app in zipped format via blackboard. Use proper coding techniques and naming conventions for objective C. Include your team member names and team name in the comments of the "main.m" file as well as in blackboard upload text. Upload as "teamNameAssignmentThree.zip".

    Remember to also fill in the team member contribution form and back up all the contributions that you assign.

    This is assignment is shorter than other assignments and worth fewer points.

    Grading Rubric
    • [10 points] Proper Interface Design
    • [10 points] Proper Coding Techniques for Swift/Objective C
    • [40 points] Module A (acitivity accessed properly, steps counted correctly, goals setting)
    • [30 points] Module B (SceneKit used properly, physics, proper motion access, etc.)
    • [10 points] Exceptional (free reign to make updates, perhaps on the interface side)

    Assignment Four - Core Image and OpenCV

    Module A
    Create an iOS application using the CoreImage template that:
    • Reads and displays images from the camera in real time
    • Highlights multiple faces in the scene using CoreImage filters
    • Highlights eye and mouth position using CoreImage filters
      • hint: could use another filter for highlights
    • an idea for exceptional work: display if the user is smiling or blinking (and with which eye)
    Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email).

    Module B
    Create an iOS application using the OpenCV template that:
    • Uses video of the user's finger (with flash on) to sense a single dimension stream indicating the "redness" of the finger
    • Uses the redness to measure the heart rate of the individual (coarse estimate)
    • An idea for exceptional work: Display an estimate of the PPG signal
    • An idea for exceptional work: Use face detection to look at changes in color of a user's face to get heart rate (i.e., Poh's method)
    Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email).

    Turn in the source code for your app in zipped format via blackboard. Use proper coding techniques and naming conventions for objective C. Include your team member names and team name in the comments of the "main.m" files as well as in blackboard upload text. Upload as "teamNameAssignmentFour.zip".

    Remember to also fill in the team member contribution form and back up all the contributions that you assign.

    Grading Rubric
    • [8 points] Proper Interface Design
    • [35 points] Algorithm Design (Efficiency and Approach for each module)
    • [27 points] Module A (faces highlighted, eyes, mouth, orientations, multiple faces)
    • [20 points] Module B (heart rate accuracy)
    • [10 points] Exceptional (free reign to make updates, perhaps on the interface side)

    Assignment Five - Bluetooth Arduino Sensing

    Create an iOS application using the Bluetooth template and the Arduino Bluetooth Shield that:
    • Reads and displays data from two or more sensors/hardware attached to the arduino
      • one sensor/input must use analog voltage (e.g., as simple as a potentiometer)
      • the other inputs(s) can be analog, digital, or binary output (e.g., as simple as a button)
      • the display of the sensor data should be more than just a text label
      • use proper sampling rates to display the sensor output
      • use proper dynamic range and voltage references
      • Create a protocol for encoding and interpretting the data on the iPhone
    • Sends two or more control commands to the microcontroller that change the behavior of the arduino (i.e., the arduino interpretes the commands and changes something)
      • make the controls change something noticeable in the operation of the Arduino
      • the output must make use of a PWM signal (implemented in hardware)
    • The Arduino sketch should also:
      • use one or more interrupts
        • for example, use a button as input
        • the interrupt must change something noticeable in the operation of the Arduino
        • use proper debouncing
      • use digital outputs (GPIO)
        • for example to control LED(s), pin 13 is already setup for this
      • use PWM (in hardware, look at which GPIO pins can do this)
      • use the ADC (for the analog input part)
    Use the tips from lecture for proper interfacing of sensors, buttons, timers, PWM, and interrupts. Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email).

    Turn in the source code for your app and Arduino sketch in zipped format via blackboard. Use proper coding techniques and naming conventions for objective C. Include your team member names and team name in the comments of the "main.m" file as well as in blackboard upload text. Upload as "teamNameAssignmentFive.zip". For this assignment, it is extremely important to show the application to me or archive it in video as I will not be able to test your hardware designs.

    Remember to also fill in the team member contribution form and back up all the contributions that you assign.

    Grading Rubric
    • [5 points] Proper Interface Design
    • [30 points] Peripherals Selected Properly, Effective Protocol Created
    • [10 points] Proper display of sensor data in a way that makes sense for the data collected
    • [35 points] Proper Use of Hardware (ADC, Interput, PWM, GPIO, etc.)
    • [10 points] Proper control commands sent and used by arduino
    • [10 points] Exceptional (free reign to make updates, perhaps on the interface or types of sensors)

    Assignment Six - Machine Learning as a Service

    Module A
    Create an iOS application using the HTTPExample that:
    • Collects some form of low throughput (sampling rate > 1s) feature data for processing, audio, video, motion, or from the arduino
    • Uploads labeled feature data to a server via HTTP POST requests
      • you can run the server from your laptop or mac mini
      • Alternatively you can use a virtual machine or AWS
    • Trains a model from the labeled data (e.g., KNN)
    • Requests predictions from the server by uploading unknown feature vectors
      • can be periodically or initiated by user
    • Note that the server code given to you will automatically save any feature data you upload and train a machine learning model, given the correct POST/GET request commands
    You should not need to update the server for any of the given functionality. However, the predictions from the server may not be sufficient without updating the training parameters or the form of model. Verify the functionality of the application to the instructor during lab time or office hours (or scheduled via email).

    You have a lot of free reign in this assignment to create something interesting and unique. Try to make this one iteration of the final project. For module B, you may choose one of two different options.

    Module B: Option 1
    Update the HTTPExample and the tornado web server to:
    • Specify the type of model to use in the Machine Learning
      • at least two different types of machine learning models (e.g., SVM and KNN)
    • Compare the efficacy of two or more different models
      • send parameters to use in the machine learning models from the phone (e.g., number of neighbors to use in KNN)
    • An option for exceptional work: make the training of the model non-blocking to the tornado IOLoop or implement authentication in tornado and in your iOS application.
    Module B: Option 2
    Use the IBM Watson knowledge engine (choosing this option auotmatically qualifies for half of the exceptional work requirement--getting it working accounts for all). Note that this is more work and requires careful collaboration with the instructor to get working.
    • Give the instructor five "cleaned" word documents that are representative of the task we discussed in class for Watson to learn from
    • Help bias the Watson eco-system by asking several example questions
    • Create a part of the app for the user to ask (or type) plain language questions
    • Query the Knowledge engine and display the results from Watson, as well as the evidence Watson used to answer the question
    Turn in the source code for your app in zipped format via blackboard (and the updated source code for the Tornado web server). Use proper coding techniques and naming conventions for objective C, swift, python, and arduino C. Include your team member names and team name in the comments of the "main.m" file as well as in blackboard upload text. Upload as "teamNameAssignmentSix.zip".

    Remember to also fill in the team member contribution form and back up all the contributions that you assign.

    The final project proposal is due at the same time as this assignment.

    Grading Rubric
    • [10 points] Proper Interface Design
    • [20 points] Design: Sufficient Calibration Protocol, Well Designed Feature Selection
    • [40 points] Module A: Features Collected, Uploaded, Predictions Made, Errors handled
    • Option 1:
      • [20 points] Module B: Opt. 1, Models Created, Accuracy of Models Compared with Different Attributes sent for Training via Phone
      • [10 points] Exceptional (free reign to make updates, authentication, more machine learning, non-blocking training, etc.)
    • Option 2:
      • [20 points] Cleaned Documents, Bias Questions
      • [10 points] Questions answered and displayed via restful API, evidence displayed

    Final Project Demonstration and KickStarter Webpage/Video

    Students will also be working on a longer final project that will culminate in a final demonstration. Students will receive guidance on the selection of a final project, but will mostly have creative free reign to develop and evaluate a mobile sensing application. The difficulty of the final project should be on par with about two lab assignments. You will be graded based on (1) the clarity of explanations of technical aspects of the application and (2) the criteria met by your original proposed specifications.

    Create an iOS application that uses the techniques for programming we learned in class. The final project should incorporate at least one technique from three of the six labs assignments. The difficulty of the final project should be at least that of two lab assignments. Ideally, your final project will build upon lab six. 

    Proposal
    All final projects should be approved by the instructor via the final project proposal. This is a description explaining the overall idea, which labs the project builds from, and a list of four or more design constraints that the design will meet. Turn in the project proposal via blackboard.

    If your group is opting into the "mother of all demos" (see explanation below) your proposal must specify this (you can opt out later, but you cannot opt in). Note that MOD projects should have more difficult constraints. The instructor may tighten constraints for those that wish to opt into the MOD.

    Demonstration and Presentation
    During final exam time, teams will demonstrate the finished application and give a brief, polished presentation of the application. The presentation should include:
    • a brief introduction of the app (~two minutes)
      • motivate why the application is useful
      • make the intro memorable using techniques we talked about in class
    • an overview of the methods used (~three minutes)
      • highlight the difficult problems that you solved and how
      • present yourself as a professional and expert (and your own personality)
    • time for demonstration and questions (~five minutes)
      • You will answer questions as posed by the instructor and the class
    The presentation should be geared toward individuals familiar with the material from this course--it does not need to be ostensible to a general audience. The presentation is worth 5% of the grade for this class.

    KickStarter / IndieGoGo Video
    You will submit a video write-up similar to vidoes one might see on kickstarter. You can think of this as a video pitch would give potential customers and investors (this should be approachable by a wider audience). It should be appealing and fun to watch, but yet informative enough to showcase your expertise in mobile programming. You should concisely explain what your product is, what value it has to offer, features, specification, and maybe even a tagline.

    Note that you are NOT being asked to host the project on kickstarter–only to develop a marketing style summary. You are not required to host the video publicly, although you are encouraged. Please submit the video via download link, youtibe link (unlisted or public) or directly to blackboard. Also submit all source code for the project via blackboard.

    The Mother of all demos (MOD)
    Students will have the option to “opt out” of the final project video in exchange for a more risky “mother of all demos” demonstration. Groups that opt into this must meet all of the proposed specifications of their mobile applications (additionally, these demos should be more difficult). These specifications must work flawlessly during the final project demonstration. If any specification is not met, the MOD does not apply and a video summary will be due. Note that it does not matter how close the specification was to working: if a portion of the application was specified to run in 5 seconds, but takes 5.1 seconds, the MOD is not met (it is all or nothing). You will be told immediately after demonstration whether the design specifications were met.

    If you succeed in the MOD, then you must still upload all source code to blackboard, but the video is not due.