AI-Powered 3D Modeling Tool Empowers Blind and Low-Vision Programmers

For blind and low-vision programmers, the world of 3D modeling has long been inaccessible. Traditional software relies heavily on visual interaction – dragging, rotating, and inspecting shapes on screen – creating a significant barrier to entry for those without sight. This limitation impacts work in fields like hardware design, robotics, and engineering, effectively excluding a talented pool of potential contributors. However, a new tool called A11yShape is aiming to change that, offering a pathway for visually impaired programmers to independently create, inspect, and refine three-dimensional models.

The challenge isn’t simply about writing code to *describe* a 3D model. While code-based modeling tools like OpenSCAD allow users to define shapes through text commands – for example, cylinder (h=20, d=5) creates a cylinder 20 units high and 5 units in diameter – blind users have historically lacked a way to verify the accuracy of their code’s output. They can write the instructions, but without visual feedback, understanding whether the shape is correct, properly aligned, or meets design specifications has been impossible without assistance from sighted colleagues.

A11yShape bridges this gap by acting as a “set of eyes” for the user. Developed by a multi-university research team including researchers at the University of Michigan, the University of Texas at Dallas, the University of Washington, Purdue University, and Stanford University, the system leverages the power of large language models (LLMs), specifically GPT-4o, and integrates with OpenSCAD. Every time a user writes OpenSCAD code, A11yShape renders the 3D model from multiple angles – top, bottom, left, right, front, and back – creating a comprehensive visual snapshot. This snapshot, along with the code itself, is then fed into GPT-4o for analysis and description.

The core innovation of A11yShape lies in its cross-representation highlighting mechanism. The system synchronizes semantic selections across all model representations – the code, a hierarchical representation of the model’s components, the AI-generated description, and the 3D rendering. If a user selects a specific line of code or a component within the model, A11yShape highlights the corresponding element in all other representations, providing a clear and consistent understanding of the relationship between code and visual output. This allows blind and low-vision programmers to independently verify their work and iterate on designs without relying on sighted assistance.

The project originated from a conversation between Liang He, assistant professor of computer science at the University of Texas at Dallas, and a low-vision classmate studying 3D modeling. He recognized the need for a practical tool based on the coding strategies his classmate had learned in a 3D modeling course for blind programmers at the University of Washington. “I want to design something useful and practical for the group,” He explained, emphasizing his commitment to creating a tool driven by the needs of the visually impaired community.

A11yShape builds upon the strengths of OpenSCAD, a script-based 3D modeling editor that eliminates the need for mouse-driven clicking and dragging. The program adds features that connect each component of the modeling process across three user interface panels: a code editor, an AI assistance panel, and a model panel displaying the hierarchical structure and rendering of the model. The AI Assistance Panel allows users to submit real-time queries to GPT-4o to validate design decisions and debug OpenSCAD scripts.

Early user testing with four blind and low-vision programmers yielded promising results. Participants were able to independently comprehend existing 3D models, create new models, and modify existing designs – tasks previously impossible without sighted assistance. One participant, new to 3D modeling, described the tool as providing “a new perspective,” demonstrating the potential for the visually impaired community to engage with 3D design.

However, the research also highlighted areas for improvement. Participants noted that lengthy text descriptions could be challenging for grasping complex shapes, and that a physical model or tactile display would be beneficial for a more complete understanding of the design. The research team also evaluated the accuracy of the AI-generated descriptions, finding that sighted participants rated them highly for geometric accuracy, clarity, and lack of “hallucinations” – inaccurate or nonsensical information.

Future iterations of A11yShape are expected to address these limitations, potentially integrating tactile displays, real-time 3D printing capabilities, and more concise AI-generated audio descriptions. Stephanie Ludi, director of DiscoverABILITY Lab and professor of computer science and engineering at the University of North Texas, emphasized the broader impact of the tool, noting that it not only supports professional programmers but also lowers the barrier to entry for blind and low-vision learners interested in exploring 3D modeling and digital fabrication. “People like being able to express themselves in creative ways… using technology such as 3D printing to make things for utility or entertainment,” Ludi said. “Persons who are blind and visually impaired share that interest, with A11yShape serving as a model to support accessibility in the maker community.”

The A11yShape project was presented at the October ASSETS conference in Denver, marking a significant step towards greater inclusivity in the field of 3D modeling and design.