Roland Bruggmann — Portfolio

Personal Pages

« Back to Main Page

Unreal Engine Plugin: Institute of Sport Science ISPW

Version: v2.0.1
Author: Roland Bruggmann

Description

Screenshot of Plugin Content

A game plugin providing with Institute of Sport Science ISPW related assets. These are mainly Blueprints inheriting from plugin ‘Distributed Game Engine DGE’ but customized to environments of the ISPW, especially the Sensorimotor Laboratory with its five sided CAVE.

Features

Contents

Dependencies

The plugin was developed using Visual Studio 2019. It makes use of code and/or assets from other plugins which must also be installed (cp. package diagram):

Package Diagram:

Package Diagram

Usage

Use the plugin as project plugin (folder MyProject/Plugins). Add the plugin by downloading and unpackaging an archive or using git clone:

git clone https://github.com/brugr9/ISPW.git

To create a new level running in the SenorimotorLab CAVE copy the template map to the project content folder.

Screenshot of Plugin


Table of Contents

1. ISPW CAVE

1.1. Handedness and Units

Handedness:

Units:

Screenshot of Project Settings, Plugin ESBMessaging:

Screenshot of Plugin Settings ESBMessaging

1.2. Real CAVE and Virtual Twin

Screenshot of CAVE model StaticMesh SM_ISPW_Cave_InverseNormals with 22 named locator sockets:

Screenshot of StaticMesh SM_ISPW_Cave_InverseNormals

2. Maps

2.1. Map ISPW Cave 3D Demo

The map ISPW_Cave_3D_Demo is intended to demonstrate the real-time rendering of 3D content in mono but also in stereo mode and the perception of the same using shutter glasses in the ISPW SenorimotorLab CAVE:

To create a new SenorimotorLab CAVE level for 3D real-time rendering demonstration copy the template map ISPW_Cave_3D_Demo into your project content folder. Please remark: The level as is does not support ESB-Messaging.

Screenshot of map ISPW_Cave_3D_Demo in the UE4 Editor:

Map ISPW_Cave_3D_Demo

2.2. Map ISPW Cave 360DegVideo

The map ISPW_Cave_360DegVideo provides with lab integration enabled by a newly developed messaging layer which allow VR-objects to be controlled by the lab’s experiment management system and CAVE-specific content to be rendered:

To create a new SenorimotorLab CAVE level supporting ESB-Messaging and 360° Video Playback copy the template map ISPW_Cave_360DegVideo into your project content folder.

Screenshot of map ISPW_Cave_360DegVideo in the UE4 Editor:

Map ISPW_Cave_360DegVideo

2.2.1 Message Transport

The map implements the ZeroMQ PUB/SUB Pattern from Plugin ‘ESB-Messaging’ using communication channels as follows:

Screenshot of Map ISPW_Cave_360DegVideo, World Outliner Folder ZeroMQ in UE4 Editor:

Map ISPW_Cave_360DegVideo, World Outliner Folder ZeroMQ

Screenshot of Project Settings, Plugin ESBMessaging:

Screenshot of Plugin Settings ESBMessaging

2.2.1.1. Communication Channel ‘Event’

Communication Diagram ‘Event’:

Communication Diagram Event

2.2.1.2. Communication Channel Datastream ‘EyeLeft’

Communication Diagram ‘Datastream EyeLeft’:

Communication Diagram Datastream EyeLeft

2.2.1.3. Communication Channel Datastream ‘EyeRight’

Communication Diagram ‘Datastream EyeRight’:

Communication Diagram Datastream EyeRight

2.2.1.4. Communication Channel Datastream ‘Head’

Communication Diagram ‘Datastream Head’:

Communication Diagram Datastream Head

2.2.1.5. Communication Channel Datastream ‘HeadProjection’

Communication Diagram ‘Datastream HeadProjection’:

Communication Diagram Datastream HeadProjection

2.2.1.6. Communication Channel Datastream ‘Watchdog’

Communication Diagram ‘Datastream Watchdog’:

Communication Diagram Datastream Watchdog

2.2.1.7. Communication Channel Datastream ‘VideoRenderTick 360Deg’

Communication Diagram ‘Datastream VideoRenderTick 360Deg’:

Communication Diagram Datastream RenderTick 360

2.2.1.8. Communication Channel Datastream ‘VideoRenderTick Planar’

Communication Diagram ‘Datastream VideoRenderTick Planar’:

Communication Diagram Datastream RenderTick Planar

2.2.2 Level Blueprint

Screenshot of map ISPW_Cave_360DegVideo Level Blueprint in UE4 Editor:

Map ISPW_Cave_360DegVideo Level Blueprint

3. Blueprints

An overview of Blueprints and handled object names (Actor-Component ESBMessageHandler, Handled ObjectName) and supported function names (ESBMessaging Interfaces) or expected responses respectively may be found in table ‘ISPW ObjectNames and Interface Functions’:

Table ISPW ObjectNames and Interface Functions

3.1. BP ISPW DGEMediaPlayer2D

See also Plugin ESBMessaging documentation, section “UE4 MediaPlayer as State Machine”.

Screenshot of BP_ISPW_DGEMediaPlayer2D in ReferenceViewer:

Screenshot of BP_ISPW_DGEMediaPlayer2D in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / BP_ISPW_DGEMediaPlayer2D):

Communication Diagram Event:

Communication Diagram Event

Communication Diagram Datastream RenderTick:

Communication Diagram Datastream RenderTick Planar

3.2. BP ISPW DGEMediaPlayer360

Screenshot of BP_ISPW_DGEMediaPlayer360 in ReferenceViewer:

Screenshot of BP_ISPW_DGEMediaPlayer360 in ReferenceViewer

3.3. BP ISPW Cave DGEMediaPlayer360

1) The screen of DGEMediaPlayer360 is a sphere. The screen sphere encases the CAVE cuboid. The sphere and the cuboid should never intersect. The center of the sphere corresponds to the position of the camera, which is moved by a MoCap tracked person inside the CAVE. Assuming that the center of the sphere is always inside the cuboid, the radius of the sphere must therefore be at least equal to the space diagonal of the cuboid. The cuboid space diagonal measures sqrt((1100.0 UU)^2 + (600.0 UU)^2 + (420.0 UU)^2) = 1321.51 UU which, rounded up, gives a value of 1400.0 UU. Used as sphere radius results in a sphere diameter of 2800.0 UU.

Example Result Messages

See also Plugin ESBMessaging documentation, section “UE4 MediaPlayer as State Machine”.

Screenshot of BP_ISPW_Cave_DGEMediaPlayer360 in ReferenceViewer:

Screenshot of BP_ISPW_Cave_DGEMediaPlayer360 in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / BP_ISPW_Cave_DGEMediaPlayer360):

Communication Diagram Event:

Communication Diagram Event

Communication Diagram Datastream RenderTick:

Communication Diagram Datastream RenderTick 360

3.4. BP ISPW DGETextRender

Please consider:

Screenshot of BP_ISPW_DGETextRender in ReferenceViewer:

Screenshot of BP_ISPW_DGETextRender in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / BP_ISPW_DGETextRender):

Communication Diagram Event:

Communication Diagram Event

3.5. BP ISPW DGEWatchdog

Screenshot of BP_ISPW_DGEWatchdog in ReferenceViewer:

Screenshot of BP_ISPW_DGEWatchdog in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / BP_ISPW_DGEWatchdog):

Communication Diagram Datastream Watchdog:

Communication Diagram Datastream Watchdog

3.6. BP ISPW Cave DGEAprilTags

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

The Blueprint makes use of CAVE model StaticMesh which provides with locator sockets, see screenshot of StaticMesh SM_ISPW_Cave_InverseNormals:

Screenshot of StaticMesh SM_ISPW_Cave_InverseNormals

Screenshot of BP_ISPW_Cave_DGEAprilTags with bundled AprilTag 36_11 Images in ReferenceViewer:

Screenshot of BP_ISPW_Cave_DGEAprilTags in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BP_ISPW_Cave_DGEAprilTags):

Communication Diagram Event:

Communication Diagram Event

3.7. BP ISPW Cave DGEColorChecker

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

The Blueprint makes use of CAVE model StaticMesh which provides with locator sockets, see screenshot of StaticMesh SM_ISPW_Cave_InverseNormals:

Screenshot of StaticMesh SM_ISPW_Cave_InverseNormals

Screenshot of BP_ISPW_Cave_DGEColorChecker in ReferenceViewer:

Screenshot of BP_ISPW_Cave_DGEColorChecker in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BP_ISPW_Cave_DGEColorChecker):

Communication Diagram Event:

Communication Diagram Event

3.8. BP ISPW Cave DGEClearColor

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Screenshot of BP_ISPW_Cave_DGEClearColor in ReferenceViewer:

Screenshot of BP_ISPW_Cave_DGEClearColor in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BP_ISPW_Cave_DGEClearColor):

Communication Diagram Event:

Communication Diagram Event

3.9. BP ISPW Cave DGEBlackShieldBack

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Screenshot of BP_ISPW_Cave_DGEBlackShieldBack in ReferenceViewer:

Screenshot of BP_ISPW_Cave_DGEBlackShieldBack in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BlackShield / BP_ISPW_Cave_DGEBlackShieldBack):

Communication Diagram Event:

Communication Diagram Event

3.10. BP ISPW Cave DGEBlackShieldFloor

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BlackShield / BP_ISPW_Cave_DGEBlackShieldFloor):

Communication Diagram Event:

Communication Diagram Event

3.11. BP ISPW Cave DGEBlackShieldFront

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BlackShield / BP_ISPW_Cave_DGEBlackShieldFront):

Communication Diagram Event:

Communication Diagram Event

3.12. BP ISPW Cave DGEBlackShieldLeft

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BlackShield / BP_ISPW_Cave_DGEBlackShieldLeft):

Communication Diagram Event:

Communication Diagram Event

3.13. BP ISPW Cave DGEBlackShieldRight

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BlackShield / BP_ISPW_Cave_DGEBlackShieldRight):

Communication Diagram Event:

Communication Diagram Event

3.14. BP ISPW DGEHead

Screenshot of BP_ISPW_DGEHead in ReferenceViewer:

Screenshot of BP_ISPW_DGEHead in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BP_ISPW_DGEHead):

Communication Diagram Event (Interface IRenderProperty):

Communication Diagram Event

Communication Diagram Datastream Head (Interface EsbTransform):

Communication Diagram Datastream Head

3.15. BP ISPW Cave DGECollisionWarning

Table ‘ISPW PlayArea-Objects Location and Scale’:

Table ISPW PlayArea-Objects Location And Scale

Screenshot of BP_ISPW_Cave_DGECollisionWarning in ReferenceViewer:

Screenshot of BP_ISPW_Cave_DGECollisionWarning in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / BP_ISPW_Cave_DGECollisionWarning):

Communication Diagram Event:

Communication Diagram Event

3.16. BP ISPW DGEPupilGaze0

Please consider:

Screenshot of BP_ISPW_DGEPupilGaze0 in ReferenceViewer:

Screenshot of BP_ISPW_DGEPupilGaze0 in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilGaze0):

Communication Diagram Event (Interface IRenderProperty):

Communication Diagram Event

Communication Diagram Datastream EyeLeft (Interface EsbTransform):

Communication Diagram Datastream EyeLeft

3.17. BP ISPW DGEPupilGaze0Ray

Please consider:

Screenshot of BP_ISPW_DGEPupilGaze0Ray in ReferenceViewer:

Screenshot of BP_ISPW_DGEPupilGaze0Ray in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilGaze0Ray):

Communication Diagram Event:

Communication Diagram Event

3.18. BP ISPW DGEPupilGaze0Hit

Please consider:

Screenshot of BP_ISPW_DGEPupilGaze0Hit in ReferenceViewer:

Screenshot of BP_ISPW_DGEPupilGaze0Hit in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilGaze0Hit):

Communication Diagram Event:

Communication Diagram Event

3.19. BP ISPW DGEPupilGaze1

Please consider:

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilGaze1):

Communication Diagram Event (Interface IRenderProperty):

Communication Diagram Event

Communication Diagram Datastream EyeRight (Interface EsbTransform):

Communication Diagram Datastream EyeRight

3.20. BP ISPW DGEPupilGaze1Ray

Please consider:

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilGaze1Ray):

Communication Diagram Event:

Communication Diagram Event

3.21. BP ISPW DGEPupilGaze1Hit

Please consider:

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilGaze1Hit):

Communication Diagram Event:

Communication Diagram Event

3.22. BP ISPW DGEPupilShapeTarget

Screenshot of BP_ISPW_DGEPupilShapeTarget in ReferenceViewer:

Screenshot of BP_ISPW_DGEPupilShapeTarget in ReferenceViewer

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilShapeTarget):

Communication Diagram Event:

Communication Diagram Event

3.23. BP ISPW DGEPupilShapeTarget HeadProjection

Customized Blueprint values in map ISPW_Cave_360DegVideo (Path DGE / PlayArea / Pupil / BP_ISPW_DGEPupilShapeTarget_HeadProjection):

Communication Diagram Event (Interface IRenderProperty):

Communication Diagram Event

Communication Diagram Datastream HeadProjection (Interface EsbTransform):

Communication Diagram Datastream HeadProjection

4. Testing

4.1 Display Cluster Configuration

The UE4 DisplayCluster aka nDisplay configuration files may be found in Folder Content/DisplayClusterConfigs/<Dev,Prod>:

nDisplay config files *.cfg (host IPs, master etc.) were edited using a text editor of choice.

File name convention: CAVE-<n>_-_3DMode-<Off,DualHead>_-_Camera-<Static,HeadTracked>_-_Projection-<Simple,EasyBlend><_-_unwrap>.cfg

The file name is explained as follows:

CAVE-<n>

n sided CAVE

_-_3DMode-<Off,DualHead>

Naming corresponding to Barco Projector settings, 3DMode:

_-_Camera-<Static,HeadTracked>

_-_Projection-<Simple,EasyBlend>

<_-_unwrap>

Config Files

Folder ‘Dev’:

Folder ‘Prod’:

4.2. Packaging

Brief instruction: Packaging, e.g., a shipping build (cp. Packaging Projects):

  1. Project Settings > Project > Maps & Modes > Default Maps > Game Default Map: ISPW_Cave_360DegVideo
  2. File > Package Project > Build Configuration > Shipping
  3. File > Package Project > Windows (64-bit)
  4. Point to a folder of your choice

In the folder of your choice a folder ‘WindowsNoEditor’ is created containing the executable file (*.exe).

4.3 nDisplay Launcher

Brief instruction: Run an ISPW-CAVE game unwrapped locally on your machine (cp. nDisplay Quick Start):

  1. Find and launch nDisplay Listener and nDisplay Launcher (folder ‘C: \Program Files \Epic Games \UE_4.26 \Engine \Binaries \DotNET’ or similar)
  2. In nDisplay Launcher, Tab ‘Launcher’, List of Applications > Click Button ‘Add’: Point to the shipping build executable ‘*.exe’ in folder ‘WindowsNoEditor’
  3. In nDisplay Launcher, Tab ‘Launcher’, Config Files Dropdown > Click Button ‘Add’: Point to a Display Config, e.g., ‘\WindowsNoEditor \SensorimotorLab \Content \DisplayClusterConfigs \Dev \CAVE-5_-_3DMode-Off_-_Camera-Static_-_Projection-Simple_-_unwrap.cfg’
  4. Click Button ‘Run’

Screenshot of nDisplay Launcher:

nDisplayLauncher

Screenshot of nDisplay Listener:

nDisplayListener

Screenshot of nDisplay Unwrap, map ISPW_Cave_360DegVideo:

nDisplayUnwrap

4.3.1 Load Non Default Map

To load a map other than the default one, e.g., map “ISPW_Cave_3D_Demo”, in nDisplay Launcher enter the custom command line argument /Game/Maps/ISPW_Cave_3D_Demo (cp. Unreal Engine Documentation, Command Line Arguments).

Screenshot of nDisplay Unwrap, map ISPW_Cave_3D_Demo:

nDisplayUnwrap3D

4.3.2 Test Pattern

To load a test pattern make use of Cluster Events (cp. Unreal Engine Documentation, Test-Patterns).

4.4 Jupyter Notebook

For testing purpose this plugin provides with Jupyter Notebooks:

Please find instructions on how to install Anaconda and setup Jupyter Notebook in folder Testing.

Screenshot of Jupyter Notebook ISPW ZeroMQ Publish:

Screenshot of Jupyter Notebook ISPW ZeroMQ Publish

A. References

« Back to Main Page