AudioCraft

This is an experimental prototype for Blind Accessibility written a few years ago to demonstrate a Sonified Depthmap Renderer with Object Filtering capabilities for navigating first person 3D spaces purely with sound, based on a Minecraft example written by Michael Fogleman. The Sonifier components are based on previous work with Peter Meijers Image To Sound algorithm. Python Source Code is included.

How it works is the engine grabs a snapshot of both the depth buffer and frame buffer, drawing solid colored textures with no fog or shading gradients. Both the depthmap and frame image are fed into the sonifier, where the frame image can be used as a stencil to cut out specific parts of the Depthmap by isolating particularly selected colors, the resulting depthmap can then be converted into sound and convey targeted objects positions over a 3D area, such as specific blocks, though could include things like specific opponents, objectives, items, interactive objects, hazards, doors, etc. Ideally the depthmap is arranged so that closer objects are brighter, with further objects darker, though adjustments to this could provide greater flexibility. This method could be applied to any existing 3D engine which makes use of depthmaps in its rendering pipeline as an available accessibility option for navigation and complement other features, such as good environmental audio, beacons, spoken menus, and cues, etc.

To play, you can use the W, A, S, D keys to move around and the arrow keys or mouse to look around. Space to Jump. Tab for flying mode. Numbers 1-3 for selecting a material to build with. Mouse left click or shift to remove a block, right click or enter to place a block. Q to play a blocks unique sound. Z for speaking what material the block is.

For the Sonifier, you can press E to scan the area in front of you, and select a filter type with the F1 to F5 keys, F6 for looping it, with F7 and F8 to increase and decrease playback speed respectively.

If you encounter any performance issues, try to Disable Display Scaling on High DPI Settings and run it in compatibility mode for WinXP SP3.