Multimodal Data Fusion for
Cyber-Physical-Human Systems
Athesissubmittedinfulfilmentoftherequirementsforthedegreeof
MasterofEngineering
Lars Joyce Planke
BEng(Electrical)(Hon1stClass)(RMITUniversity)
SchoolofEngineering
CollegeofScience,Technology,EngineeringandMaths
RMITUniversity
May2021
I
Declaration
Icertifythatexceptwheredueacknowledgementhasbeenmade,thisworkisthatofthe
authoralone;thisworkhasnotbeensubmittedpreviously,inwholeorinpart,toqualifyfor
anyotheracademicaward;thecontentofthethesisistheresultofwork,whichhasbeen
carriedoutsincetheofficialcommencementdateoftheapprovedresearchprogram;any
editorialwork,paidorunpaid,carriedoutbyathirdpartyisacknowledged;and,ethics
proceduresandguidelineshavebeenfollowed.
IacknowledgethesupportIhavereceivedformyresearchthroughtheprovisionofan
AustralianGovernmentResearchTrainingProgramScholarship.
LarsJoycePlanke
20May2021

II
Acknowledgements
InconductingmyMasterofEngineeringresearchprogram,Iwouldfirstlyliketoexpressmy
gratitudetomysupervisorsProf.Roberto(Rob)SabatiniandDr.Alessandro(Alex)Gardi,in
whichtheiradvice,guidanceandaspirationshavebeenofutmostimportanceincompleting
asuccessfulresearchproject.IwouldliketogivemythankstoRobforidentifyingapotential
inme,openingupinvaluableopportunitiesandaspiringmetostriveforexcellence.Iwould
alsoliketothankAlexforprovidingtimeandthoughtfuladvicethroughouttheproject.
AspecialthanksgoestomyfriendsandcolleaguesatRMITUniversity,includingYixiangLim,
NichakornPongsarkornsathien,SamHilton,RohanKapoor,SurajBijjahalliandFederico
Rivalta.Theiraccompanyingpresencehavemadethisendeavourhighlyrewarding,with
stimulatinginteractionsthathavesparkedvaluableinsightsandhelpedprogressmy
research.
IwouldalsoliketogivemanythankstoDr.NetaEzerfromNorthropGrummanCorporation
forprovidinghelpfulfeedbackandsupportingthisresearchproject.Also,thankstoTrevor
KistanfromThalesAustraliaforhelpingmeconductthisresearchandprovidingfeedback.I
wouldalsoliketoacknowledgeandexpressmygratitudetoRMITUniversityforselecting
meforaRMITResearchStipendScholarship.
Thisresearchprojecthasbeenconductedamidahistoricpandemic,andaparticular
gratitudegoestomypartnerBellaReidandherfamilyforprovidingextrasupportthrough
unusualanduncertaintimes,withoutwhichthisresearchwouldnothavebeenas
successful.Finally,Iwouldtoliketogivemyheartfeltappreciationtoallmyfriendsand
familyinNorwayandAustralia.NomatterthephysicaldistanceIcanalwaysrelyontheir
encouragement,careandconstantsourceofsupport.
III
Table of Contents
DeclarationI
AcknowledgementsII
ListofFiguresVIII
ListofTablesXI
ListofAcronymsandAbbreviationsXII
ExecutiveSummary1
Background______________________________________________________________3
Researchgaps____________________________________________________________6
1.2.1.Scope_______________________________________________________10
Researchaimandquestion_________________________________________________11
Researchobjectives_______________________________________________________12
Overviewofresearchmethodology__________________________________________13
1.5.1.Generalprojectmethodology____________________________________13
1.5.2.Experimentalactivities_________________________________________14
Thesisoutline____________________________________________________________16
References______________________________________________________________17
Introduction_____________________________________________________________23
Humancentredsystems___________________________________________________24
Architectureforfutureaerospacesystems____________________________________28
2.3.1.HMI2intheCNS+Acontext______________________________________28
2.3.2.AdaptivesystemsandCHMS_____________________________________30
2.3.3.CHMSinfutureaerospaceapplications____________________________32
2.3.4.Evolutionarypathsforfuturemultidomaintrafficmanagement_________37
Cognitivemodellingandmeasurements______________________________________38

IV
2.4.1.Conceptualrelationshipbetweentaskload,performanceandMWL_____38
2.4.2.Cognitivemodellinganddefinitions_______________________________41
2.4.3.TypesofmethodsformeasuringMWLandgeneratingtaskloads_______43
SensorsandmethodsforrealtimemeasurementofMWL________________________49
2.5.1.Electroencephalogram(EEG)____________________________________49
2.5.2.Eyeactivitytracking____________________________________________62
2.5.3.Cardiacactivity_______________________________________________64
2.5.4.Controlinputmeasurements____________________________________66
2.5.5.OthersensormeasurementsofMWL______________________________67
2.5.6.Multimodalityfusion___________________________________________68
2.5.7.Generalisedinferencemodels___________________________________78
Conclusions_____________________________________________________________80
References______________________________________________________________82
Introduction_____________________________________________________________95
Designconsiderationsforsensingandestimation_______________________________96
3.2.1.Toplevelrequirementsforsensingandestimation___________________97
3.2.2.Processesforsensingandestimation______________________________98
Researchmethodology___________________________________________________101
3.3.1.Generalprojectmethodology___________________________________102
3.3.2.Taskloadgeneration__________________________________________104
3.3.3.Participantinclusioncriteriaandethics___________________________109
3.3.4.Sensorequipment____________________________________________110
3.3.5.Softwareinfrastructure________________________________________115
3.3.6.Dataanalysis________________________________________________117
Experimentalapproachanddesign__________________________________________119
3.4.1.ExperimentalActivity1________________________________________119
3.4.2.ExperimentalActivity2________________________________________122
3.4.3.ExperimentalActivity3________________________________________126
Conclusion_____________________________________________________________128