From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation

Author email: suting@ntu.edu.sg
Tool name: ui2code
Description: A GUI skeleton is the starting point for implementing a UI design image. To obtain a GUI skeleton from a UI design image, developers have to visually understand UI elements and their spatial layout in the image, and then translate this understanding into proper GUI components and their compositions. Automating this visual understanding and translation would be beneficial for bootstraping mobile GUI implementation, but it is a challenging task due to the diversity of UI designs and the complexity of GUI skeletons to generate. Existing tools are rigid as they depend on heuristically-designed visual understanding and GUI generation rules. In this paper, we present a neural machine translator that combines recent advances in computer vision and machine translation for translating a UI design image into a GUI skeleton. Our translator learns to extract visual features in UI images, encode these features' spatial layouts, and generate GUI skeletons in a unified neural network framework, without requiring manual rule development. For training our translator, we develop an automated GUI exploration method to automatically collect large-scale UI data from real-world applications. We carry out extensive experiments to evaluate the accuracy, generality and usefulness of our approach.
Bibtex: @inproceedings{10.1145/3180155.3180240, author = {Chen, Chunyang and Su, Ting and Meng, Guozhu and Xing, Zhenchang and Liu, Yang}, title = {From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation}, year = {2018}, isbn = {9781450356381}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3180155.3180240}, doi = {10.1145/3180155.3180240}, booktitle = {Proceedings of the 40th International Conference on Software Engineering}, pages = {665–676}, numpages = {12}, keywords = {reverse engineering, user interface, deep learning}, location = {Gothenburg, Sweden}, series = {ICSE ’18} }
Link to public pdf: https://dl.acm.org/doi/10.1145/3180155.3180240
Link to tool webpage: http://tagreorder.appspot.com/ui2code.html
Link to demo: Not provided by authors
Category: None
Year and Conference: 2018, ICSE
Terms of use