README, Dataset Overview¶

This document serves as a readme for loading two different versions of the SMU-Textron Cognitive Load (SMU-TexCL) dataset. Specifics of this data collection are available in the methods section of the following paper:

  • [1] "Generalizing Classification of Pilot Workload: Transfer Learning versus a JEPA-Inspired Transformer Architecture" by Barnett, et al., Published in IJAAA, available at: https://commons.erau.edu/ijaaa/vol12/iss1/2/
  • [2] Features derived for processing are described in the paper: "Objective Measures of Cognitive Load Using Deep Multi-Modal Learning: A Use-Case in Aviation" by Wilson et al., published by IMWUT, available at: https://dl.acm.org/doi/abs/10.1145/3448111

When using this dataset, please cite the papers above. The dataset contains biometric information for a pilot as they fly through varying levels of turbulence in a simulator [1]. Numerous biometrics and features derived from the biometric data are available. Transfer learned features from the Wilson et al. model (BM3TX) are also included. Please see an explanation of these features in the paper referenced above [2].

Subjective scoring from the subjects for each trial is available. The cognitive load is captured using the NASA-TLX. The objective of this dataset is to use the biometric sensor streams to predict the reported task workload. This task workload has been processed in several different ways. One can use any of these measures as the ground truth label of workload. These measures are described more fully below.

The biometric data was collected on the throttle hand using an Empatica E4 sensor.

The dataset can be loaded in two forms:

  • Table Data: This has been already preprocessed such that each row in the Table represents one trial of one pilot. The features over the duration of the flight have been aggregated in various ways. These features are described more fully below.
  • Windowed Samples: This form of the dataset contains more detailed information and raw biometrics samples for each pilot and each trial. Some pre-processing of the windows has also been applied to provide addtional features. These are described more fully in the second part of this notebook.

Loading Table Data¶

The table data for this dataset has already gone through several preprocessing steps. The NASA TLX labels are also available in several different formats.

Meta Data: For each row the following information is included:

  • pilot_id: This is a randomly assigned identifier for the pilot that is flying. This code is a three digit integer. Codes with a leading 8 (i.e., 8xx) are the identifier for pilots with only minimal experience flying in a simulator. Codes with a leading 9 (i.e., 9xx) are the identifier for pilots that typically fly commercial planes. These pilots do not typically have air force training. All other codes are from pilots that are familiar with the flight simulator and have experience flying in non-commercial aircraft.
  • trial: Possible values are "Phase 1" and "Phase 2". All pilots repeated the experiment in two phases (when possible). One phase of data collection means that they flew in trials with all turbulence variations.
  • turbulence: The turbulence level for this trial. Possible values are 0, 1, 3, 5, and 7. An increasing value means more turbulence.
  • date: A timestamp for when the data collection began. The format is YYYY-MM-DD hh:mm:ss.
  • start_order: Turbulence values for each trial was randomized. This variable shows the order in which this trial was completed. This can be used for understanding an ordering effects in the data.
  • start_time, end_time, and duration: These variables contain timing information about the trials, in seconds. Start time is the number of seconds since the start of the entire experiment to the start of this trial. End time denotes the number of seconds since the start of the entire experiment to the end of this trial. Duration is the total number of seconds for this trial. Most trials are about 70-90 seconds in duration.
  • turbulence_check, augmented: These are internal error checking attributes and can be ignored. Augmented should always be false and turbulence_check should alway match the turbulence attribute.

Cognitive Load Labels: When loading the table data the following columns are available:

  • avg_tlx: This is the raw NASA-TLX score from the participant on a scale of 0-100. That is, this is the result of averaging across all the NASA-TLX scores with equal weight to arrive at one measure of the cognitive load index.
  • mental_effort': This is the 0-100 score from the category "Mental Effort" in the NASA-TLX. This is sometimes used as a proxy for cognitive load, rather than the raw NASA-TLX average score.
  • rating: This is the rating assigned to the participant based upon their reported TLX and the experimental observer. Possible values are 'Low','Medium', 'Somewhat High', 'High', and 'Very High'
  • avg_tlx_zscore: This is a processed version of the avg_tlx attribute label. A z-score is calculated based upon the mean and standard deviation of all values for this pilot. In this way, this value represents how loaded the pilots was based upon their other reported values.
  • avg_mental_effort_zscore: This is a processed version of the mental_effort attribute label. It is calculated similarly to the avg_tlx_zscore described above.
  • avg_tlx_quantile: This method transforms a pilot's scores into two quantiles, assuming that the avg_tlx should follow a normal distribution for each pilot. These transformed values should therefore align across pilots, showing a normal distribution.
  • tlx_dynamic_range: For each pilot, this is the difference between the minimum and maximum reported avg_tlx values. This value is the same for all entries for a particular pilot.

Feature Data: The following precomputed features are available: -bm3tx_0 to bm3tx_31: These 32 features represent the bottleneck features form the BM3TX model published by Wilson et al. [2].

  • sdrr: This is an estimated value of the SDRR using the raw PPG data. SDRR is the standard deviation of all RR intervals (the time between heartbeats).
  • pnn50: A meaure derived from the PPG signal. It is calculated by dividing the number of pairs of successive NN intervals that differ by more than 50ms (NN50) by the total number of NN intervals, expressed as a percentage.
  • The remaining features have statistical aggregations for each of them. For the trial of a pilot, we calculate the mean, minimum, maximum, and standad deviation of the tiem series over the duration of that trial. These statistics are calculated for:
    • eda: Electro-dermal activity.
    • temp: Wrist temperature against skin.
    • accel: Magnitude of acceleration from wrist movement on the throttle hand.
    • scr: The low freqiency component of the EDA, often referred to as the tonic response.
    • ibi: Inter-beat interval as measured by the Empatica E4 sensor.
  • scr_count: the number of peaks found in the high frequency portion of the EDA (oftern referred to as the phasic portion of the EDA signal).
In [33]:
import pandas as pd

# the table data can be loaded either from CSV or from the parquet files
file_path = 'pre_proccessed_table_data.parquet'
df = pd.read_parquet(file_path)


# using different features
feature_columns = [
   'bm3tx_0', 'bm3tx_1', 'bm3tx_2', 'bm3tx_3', 'bm3tx_4', 'bm3tx_5',
   'bm3tx_6', 'bm3tx_7', 'bm3tx_8', 'bm3tx_9', 'bm3tx_10', 'bm3tx_11',
   'bm3tx_12', 'bm3tx_13', 'bm3tx_14', 'bm3tx_15', 'bm3tx_16', 'bm3tx_17',
   'bm3tx_18', 'bm3tx_19', 'bm3tx_20', 'bm3tx_21', 'bm3tx_22', 'bm3tx_23',
   'bm3tx_24', 'bm3tx_25', 'bm3tx_26', 'bm3tx_27', 'bm3tx_28', 'bm3tx_29',
   'bm3tx_30', 'bm3tx_31', 'raw_eda_max', 'raw_eda_min', 'raw_eda_mean',
   'raw_eda_std', 'scr_count', 'scr_max', 'scr_min', 'scr_mean', 'scr_std',
   'temp_max', 'temp_min', 'temp_mean', 'temp_std', 'accel_max',
   'accel_min', 'accel_mean', 'accel_std', 'ibi_max', 'ibi_min',
   'ibi_mean', 'sdrr', 'pnn50'
]

X = df[feature_columns]

# here is an example for getting labels from the quantiles
label_column = 'avg_tlx_quantile'
y1 = df[label_column].round()

# alternatively, you could predict another attribute
label_column = 'avg_tlx_zscore'
y2 = df[label_column].round()

# these features and labels can now be used for training and testing a classifier, regressor, etc.

Loading Time Series Windows¶

This style of loading data is meant for users that wish to have access to the raw biometric signals and some precomputed values within each window of data.

The data can be loaded for each pilot and is stored in JSON format. Each window contains 60 seconds of biometric data and windows are calculated with 90% overlap to the next window. As such, there should be significant redundancy in the windows of data.

In the example below, we are loading the data for one pilot into the variable pilot_data. This variable is a list, where each element in the list corresponds to a trial. We load the trial data in the $0^{th}$ index into a new variable one_trial. Each trial is a dictionary with the following keys:

  • windowed_features: A list of the raw feature data for various overlapping windows. More information about this structure is given below.
  • meta_data: All meta data for a pilot that includes similar information to the row of table data for this pilot in this trial (see the Cognitive Load Labels section above for the Table Data).
  • label: This is the average tlx value that should be predicted from the various windows of data.

Windowed Features: We then save the various windows of data into the variable windows_of_data. For this example, there are 13 overlapping windows of data. This is a list of the data for each window. One element in the list is a dictionary of the biometric data with the following keys:

  • 'timestamp': The time, in seconds, since the beginning of experimental data colelction for this pilot. The exact time is when the window of data ends. Timestamps for subsequent windows should always be increasing by about 6 seconds.
  • 'ppg_input': The raw PPG data of a window, sampled at 64Hz. PPG stands for photoplethysmography and is a measure influenced by blood pressure as the heart beats.
  • 'eda_input': Electro dermal activity (skin conductance on the wrist) sampled at 4Hz
  • 'tonic_input': low frequency component of the EDA, 4Hz sampling rate.
  • 'accel_input': Magnitude of wrist accelration sampled at 32Hz.
  • 'temp_input': wrist temperature sampled at 4Hz
  • 'eng_features_input': 22 calculated features from the window of data. These features are normalized by sclaing algorithm that is trained from another dataset of E4 biometric data. The normalization is based on a mean and variance normalization across many hours of collected E4 data.
  • 'raw_eng_features_input': The same 22 features without normalization.
  • Note: the actual vectors that store the engineered features are longer than 22 elements, but these values are always zero and can be ignored.
    • The raw and engineer features saved correspond to previously discussed values: 'raw_eda_max', 'raw_eda_min', 'raw_eda_mean', 'raw_eda_std', 'scr_count', 'scr_max', 'scr_min', 'scr_mean', 'scr_std', 'temp_max', 'temp_min', 'temp_mean', 'temp_std', 'accel_max', 'accel_min', 'accel_mean', 'accel_std', 'ibi_max', 'ibi_min','ibi_mean', 'sdrr', 'pnn50'
In [64]:
import json

pilot_filename = 'ID001.json'

with open(pilot_filename) as fid:
    pilot_data = json.load(fid)
    
print(f'Found pilot data for pilot id: {pilot_filename} with {len(pilot_data)} numbers of trials.')

# get data for one trial
one_trial = pilot_data[0]
avg_tlx_label = one_trial['label']
windows_of_data = one_trial['windowed_features']
duration = one_trial['meta_data']['duration']

print(f'Found pilot data of {duration} seconds, with TLX value of {avg_tlx_label}.')
print(f'Found {len(windows_of_data)} windows of data.')

for one_window in windows_of_data:
    print('Seconds when window ends:', one_window['timestamp'], end=' ')
    print('duration of window', len(one_window['ppg_input'][0])/64, 'seconds')
    
    # this window of data might be processed by an algorithm or as the input to a sequential network
    # such as an LSTM or transformer. Various other forms of processing might need to be done. 
Found pilot data for pilot id: ID001.json with 10 numbers of trials.
Found pilot data of 80.45082000000548 seconds, with TLX value of 10.0.
Found 13 windows of data.
Seconds when window ends: 215.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 221.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 227.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 233.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 239.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 245.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 251.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 257.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 263.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 269.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 275.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 281.99999999999994 duration of window 60.0 seconds
Seconds when window ends: 287.99999999999994 duration of window 60.0 seconds
In [ ]:
 
In [ ]:
 
In [ ]: