Tag Archive for project

Week 3 – The State of the Union

This week was mostly spent on further research into UI.  I thought I’d take this opportunity to present a screenshot of where the VISTAS UI is at today.  It is currently a “bare metal” design; functionality has been kept at a minimum while the rendering engine has been worked on.

Vistas Screenshot

The Vistas UI, as it stands today.

Upon recommendation from Judy, I took a look at another visualization program called VisIt.  VisIt has been under development for many years by the Department of Energy, and in that time has become very feature-rich.  As you can see from the following screenshot, it is anything BUT “bare metal”:

VisIt Screenshot

A screenshot of VisIt, another visualization program.

As the scope of VisIt is presently much larger than VISTAS, I highly doubt VISTAS will ever need anything close to that many UI elements.  Nonetheless, I thought VisIt had some interesting features.  Namely:

  • Separate windows for loading/project management & visualization
  • Axis locking and zoom controls
  • Ability to change colors used dynamically via color table
  • Camera views – ability to lock, save & load
  • Save/Print Window for easy export of imagery

I also read an interesting paper: Current practice in measuring usability: Challenges to usability studies and research, Kasper Hornbaek, International Journal of Human-Computer Studies, Int. J. Human-Computer Studies 64 (2006) 79–102.  In it, Hornbaek acknowledges and evaluates the efficiency of various metrics for usability.  Particularly, he identifies the following measures of usability which can prove useful to my work on VISTAS:

  • Binary Task Completion — did the user finish their task?
  • Accuracy — how many errors did the user make while working towards their task?
  • Recall — how much information can the user remember after using the program?
  • Completeness — how complete of a solution did the user achieve?
  • Quality of Outcome — the user’s subjective satisfaction with the end result of program use

The differences between Accuracy, Completeness, and Quality of Outcome are subtle but very important.

In other project news, we are having a VISTAS teem meeting in Corvallis next week.  I am excited to visit with everyone, and am looking forward to the opportunity to talk to Margaret Burnett, an OSU professor who is an expert on UI and does some really cool research into machine learning techniques applied to UI.

Week 2 – The Spy++ Who Loved Me

This week, I learned about a fantastic tool integrated into Visual Studio called Spy++.  Many people are (painfully?) familiar with this tool, but it is new to me.  Spy++ allows you to target a UI component in a GUI application, and it will then show you a stack trace of that component.  It has already helped me a ton in figuring out how the VISTAS UI code works, as I can identify specifically what class each UI component represents and jump right to the code which generates it.  

Spy++ UI Trace

Just a taste of Spy++.

I also sat down with the application and had another brainstorm session on desired features that are within the scope of this project, which I’ll enumerate here.


Short term:

  • Fix the “Add” buttons on the Data & Visualizations tabs. Right-clicking is not intuitive to a new user. This could be simple as a small “+” or “Add” button next to
  • Project Visualizations/Project Data.
  • Add ability to delete items from Data & Visualizations tabs. Currently, removing a scene requires rebooting VISTAS.
  • Cleanup of VISTAS .ico file — remove white pixels from edge of image to beautify
  • Change default icon in titlebar of main window frame to be the VISTAS icon
  • Tooltips/mouse hover context information need to be added across the board.

Medium term:

  • Add “Help” section with integrated quick start guide
  • Revamp the Tool Bar. Re-evaluate overall layout, icon choices and functionalities present.

Longer Term:

  • Revamp the Menu Bar. Right now there is a bare minimum of functionality here, and several existing buttons don’t work.
  • “About” as its own item, and not nested under File — to be discussed.
  • Functionality to Save & Open projects needs to be implemented.  Nik has assigned himself to this.
  • Functionality to close the current project/create a new project needs to be implemented.
  • Adding/removing Viewers could perhaps be done in a more intuitive way.
  • Bundle a small sample data set with VISTAS so that a new user can experiment with the program without committing to downloading a large data set & going through all the steps of setting up the visualization. Ideally this would come in the form of a sample project include with VISTAS, so the ability to Save & Open projects needs to be implemented first.
  • Ability to take a screenshot of the viewer window, either via right-click or perhaps a button in the toolbar. Ability to copy screengrabs to clipboard for easy insertion into documents/emails/whatever.
  • Optional camera controls on the viewer window might be nice. At least a zoom control– Shift+mouse zooming is not always intuitive nor precise for fine camera movements.
  • Ability to click on colors in Legend and change them.

— whew!  I better get to work.

Week 1 – Getting Set Up

I spent this week cloning the latest VISTAS build off of the bitbucket repo and making sure that my Visual Studio can successfully build it.  This was a bit tricky, as large open source software projects inevitably gather a lot of dependencies, but I believe that I have my workflow in place.

I also set up this blog, submitted my project proposal (see previous post) and did some online resource hunting.  I’ll list these resources here for me to come back to…

VISTAS Related:

VISTAS Project Blog
VISTAS Source Repository

wxWidgets Related:

wxWidgets Alphabetical Class Reference
wxWidgets Guides & Tutorials
wxFormBuilder – Open Source wx Designer

Research Papers of Interest:

User Interface Design with Visualization Techniques – A pretty basic paper, but introduces an interesting scoring rubric for UI analysis
Current practice in measuring usability: Challenges to usability studies and research – More interesting thoughts on UI quality heuristics

Project Proposal – User Interface Developer Immersion

Summary & Learning Objectives

For my senior project, I will be working collaboratively on a small development team working on VISTAS, a 3-D scientific data visualization tool.  This project is taking place under the mentorship of Judy Cushing and is funded by the National Science Foundation.  VISTAS has already been under development for several years, so a big challenge of my project will be learning my way around an existing large C++ codebase.  The rendering engine in the application is mostly complete; however, the tool is currently slightly buggy and can be somewhat unintuitive to use.

My role will be to focus on learning user interface design & programming with the goal of enhancing the usability of the program.  I also hope to learn about modern source control methods as well as obtain practice at working on collaborative software projects.  Throughout the course of my project, I will also be fulfilling additional developer responsibilities within the VISTAS team, including attending weekly online meetings and contributing to code reviews, bug fixes, and collaborative readings of select research papers.


The existing VISTAS UI uses a cross-platform library called “wxWidgets” for all of its graphical UI elements.  There are official wxWidgets tutorials as well as a reference page online– these will be my primary resources for this project.

VISTAS-related resources can all be found at the project’s webpage.

Implementation Plan

My plan initially is to target making addition and removal of visualization scenes more intuitive.  I also hope to add context-driven mouseover tooltips for user interface elements and a basic in-application help menu complete with a quick-start guide.

When it comes to exactly what I mean by “making addition and removal of scenes and visualizations more intuitive”, I had in mind:

  • Add ability to delete items from Data & Visualizations tabs.  Currently, removing a scene requires rebooting VISTAS.
  • Add “Add” buttons to the Data & Visualizations tabs.  Right-clicking is not intuitive to a new user.  This could be simple as a small “+” or “Add” button next to Project Visualizations/Project Data.

Beyond this, I’ve compiled a list of some user quality-of-life features that I think would be good additions.  The big two in my mind are the ability to Save/Open/Close projects and the ability to add annotations/pins to visualizations, but I’d like to work up to these.

Evaluation of Work

All of my code commits to the project can be viewed at my bitbucket profile (https://bitbucket.org/jmangue/profile/activity).  Furthermore, I will be keeping weekly online journals detailing my work on VISTAS.

Project Summary: Natural Language Processing With Prolog

For my “Computing Practice and Theory” project, I chose to continue some of the work with Prolog and EBNF grammar parsing that I had begun in the “Computability” program.  Specifically, I wanted to study more about Natural Language Understanding– the process of gleaning intended meaning from natural language.

As Natural Language Processing can be a bit dry, I decided to put a fun spin on the project; I designed and implemented a small command grammar for a Scribbler II robot.  The Scribbler II is a simple educational robot which has a python-based interface library called Myro available.

Scribbler with Fluke

My program, tentatively titled “ScribPro”, is a hybrid python/prolog application that allows control of a Scribbler via natural language.  All communication with the robot is transmitted over Bluetooth.  The python component handles initialization, messaging to & from Prolog, code execution, and both fetching and sanitization of user input.  I was learning python on the fly as I did this project, so the final python component is pretty messy… but by the end of the quarter I felt that I have achieved basic competency.  I am very glad that I learned it; the ability to use python quickly create simple programs or prototype has already proven itself invaluable.

The Prolog component handles the bulk of the parsing work.  It reads the sentence fed to it from the python component, determines whether or not it is a successful parse, and if it is, converts the parse tree into a functor which it passes back to python.  The validity of a parse is determined according to a grammar.  Early on in the project, I tried to write a very abstract grammar using only the rules of the English language, and quickly realized that it was beyond the scope of this project.  This dilemma is one that anyone trying to do natural language parsing inevitably encounters: English is a messy language!  In order to overcome this and make a usable program within a ten-week timeframe, I chose to severely bound the words and sentential forms that my program would recognize.  Limiting the scope of my grammar to natural language units that are applicable to robot control made the project feasible.

While the final grammar never grew quite as complex as I would’ve liked, I am pretty happy with how the project turned out.  The robot is able to parse & execute a variety of movement-based commands — from plain English!

Not only is the robot able to handle simple sentences such as “Please go forward”, it can handle fairly complex compound sentences such as:

  • “Spin in a circle to the left two times.”
  • “Go forward four point five feet and turn left, then wait nine hundred milliseconds, turn around and beep once.”
  • “Scoot forward until you reach the wall, then move backwards two inches, turn around and beep.”

Beyond the basic text mode, there is an optional voice input mode that taps into the Microsoft Speech Engine.  It is pretty fun to vocally order the robot around!

The fact that I was using two programming languages in tandem (prolog & python) presented some significant initial difficulties in the project.  A good portion of my final code revolves around passing messages back and forth between the two.  The tradeoff was worth it, however– had I been forced to implement my program in a single language, it would have been quite a bit messier.  There is something to be said for using the right tool for the job.  By the end of the project, I gained a substantial amount of confidence in my own ability to multiple languages within a single project.  No doubt this will come in handy in the future.

Here is a diagram of the program flow and component design of my final program:

Program Flow

I now feel confident that I have a handle on the basics of natural language parsing using grammars, and believe that I could implement such a system again in a much shorter timeframe, with a more elegant end result.

If I were to revise this project, I would attempt to abstract the code-generation side to be a level higher.  Right now, the code generated from a successful parse is extremely Scribbler-specific… which I would like to rectify.  I could conceivably create my own generic set of common robotics-related functions which the parser would invoke, rather than having it call specific Myro functions.  I would then implement the ability to map these generic functions to model-specific ones via config files; essentially allowing the parser to be used for other types of robots with ease.

Overall, I had quite a bit of fun with this project, whilst still managing to feel like I challenged myself.  I look forward to further independent research into Prolog, as I feel that logic programming is a valuable yet underutilized programming paradigm.  I also look forward to further tinkering with more advanced robotics projects.

Past weekly blog updates can be found at the following links:

My final code can be found here:

  • scribpro.py – Python Main Program Executable
  • nlp.pl – Prolog Natural Language Parsing Component

(Please be forgiving of the code, as I am an undergrad and was learning by doing for most of it.  There are approximately a million things I would change if I could do it all over again.)

If anyone reading this has any questions or is working on a related project, I’d love to hear from you!

Week 9: Cleaning House (And Code)

This week, I began wrapping up my project.  Only one week remains until this thing must be worthy of demonstration in front of a class full of peers, so adding new features has to be de-emphasized in favor of reinforcing stability & adding documentation to the project.

Major New Features:

  • Added the ability to give a conditional statement.  For now, this is limited to wall sensing.  So now, “go forward until you hit a wall” is a legit command.
  • Added the ability to repeat a command a specified number of times.  i.e. “Turn left ninety degrees twice” or “Beep six times.”

Quality of Life Improvements:

  • I reorganized quite a bit of the python code and added docstrings to all the functions.
  • I added the ability to toggle voice/text input/output modes as well as debug mode.  Previously, these were command-line options or hardcoded.
  • Added punctuation filtering to user text input in order to avoid ugly crashes.  Should be a lot more user-proof now.
  • I added some more documentation to my prolog code.
  • Stabilized the photo-taking code.  It still crashes in IDLE, but works great via console python.
  • MANY small grammar reinforcements & bug fixes.

The last major feature I hope to add is a “help” menu, which will give the user an overview of commands and some examples of usage.

In the meantime, I am beginning to brainstorm about my in-class presentation and am also working on outlining my final project paper.

Next week I will post all of the sources for the final program along with a project summary.  In the meantime, here is a writeup of the full command grammar that the robot accepts:

Read more

Week 8: You Can Walk The Walk, Now Talk The Talk

I accomplished two tasks this week… adding more robustness to the movement grammar, and integrating speech recognition.

In particular, I added the abilities to parse commands such as:

  • “go left/right” — turns left 90 degrees and then moves the given distance.
  • “wait” — pauses the robot for x seconds.
  • “stop” — stops all movement.
  • “turn around” — turns 180 degrees, optionally “to the left”/”to the right”.
  • “spin around” — turns 360 degrees, optionally “to the left”/”to the right”.
  • “take a picture” — takes a photo using the fluke camera, which is transmitted to the computer via bluetooth and displayed.  This functionality is still a little buggy, which I suspect is due to some instability with my version of the Python Imaging Library.

I also loosened up the grammar in some other places to make it more flexible.  I wrote the following DCG predicate which has really come in handy:

optional(X) –> [X] ; [].

This is a case of something so obvious, that I don’t know why I didn’t think of it earlier.  That simple little line of Prolog code enables the insertion of “optional” words or phrases in a sentence and has become invaluable.  It is analogous to square-bracket notation in EBNF grammar definitions.

Integrating the Microsoft Speech Engine with the program proved to be quite a bit easier than I had imagined.  I was able to do this via a library called PySpeech.  After running through only one round of Microsoft’s voice training program, the engine is fairly accurate and fast enough to be useful.  So now, the parser can be run in either text or speech mode– pretty cool!  It is a blast to call out voice commands and see the scribbler react.

Next week I plan to wrap up the feature side of the project by adding a few more fun things (“beep” springs to mind) as well as conditional statements (“go forward until you hit a wall”).  All that will then remain is to flush out my comments and documentation.

Week 7: Scribby’s First Steps

This week, I worked solely on the Prolog natural language parsing component of the project.  This is basically a set of grammar rules along with very simple code-generation based on the results of the parse. I made quite a bit of progress this week… Basic movement commands are now parsed, along with quantities and conversions of a handful of different units. For example, you can now tell the Scribbler such things as:

* go forward for 3 seconds and then turn left
* drive forward three feet and then turn right 180 degrees
* move backwards 6.5 inches

And so on and so forth. Here is a sample run:

Enter a command: go forward two feet, turn right 90 degrees, drive backwards for three seconds
Prolog query executed: parseToCode([go,forward,two,feet,and,turn,right,90,degrees,and,drive,backwards,for,three,seconds], Status, CodeOut).
Status: valid
CodeOut: [‘forward(1.0, 4.1)’, ‘turnRight(1.0, 0.7875)’, ‘backward(1.0, 3)’]

A big component that I had to work on this week was english -> integer translation. It actually requires quite a bit of code to convert an English phrase like “nine hundred forty two point threee” to “942.3”!

My next task is to add some “fun” commands to the program; beeping, taking pictures, and maybe a wander/doodle capability.  I would also like to add some flavorful error messages to return upon a failed parse.

Another potential goal is to add conditional statements to the parser, i.e. parsing of a statement like “go forward until you hit a wall and then turn left and take a picture.”

Here is the source code for the Prolog parser as it stands today…

Read more

Week 6: Finishing the Python/Prolog interface

This week I finished the python script that handles initialization, user input, and all messaging to & from Prolog. All that is left to do is the natural language parser component in Prolog.

I was learning Python as I wrote this, so a lot of this code could be certainly written in a more elegant way. But it gets the job done.

The biggest obstacle was a silly grammatical problem I encountered was interpreting the Prolog list of code to be executed into Python code. A typical prolog code result arrives in list format [forward, ‘1.0’, ‘3.0’]. I was trying to parse that into Python as forward(1.0, 3.0). Notice the lack of string quotes around Prolog’s forward; this is because in Prolog, its data type is “atom” rather than “string”.  But this caused no end of trouble when pulling it into Python, as it was interpreting forward as a functor rather than as a string.  No problem, right?  It should be easy to just cast it as a string in Python.  But despite all my efforts, it wouldn’t work…  casting it was yielding a big mess, since it was interpreting it as a function pointer. I ended up having to resolve this issue on the Prolog side with a bit of a hack to get quotes around outgoing atoms.

Anyways, here is the Python code.  The main routine parses user input, passes it to Prolog, and then checks for a solution and returns a PrologResult instance with fields prologcall, status, and pythoncode.  Based on the result of status, it then executes pythoncode.

A typical command of “scribbler move forward” might yield the following PrologResult:

  • prologCall = parseToCode([scribbler,move,forward],Status,CodeOut).
  • status = valid
  • pythoncode = [forward(3.0, 1.0)]

Read more

Week 5 – Adventures in Integration

I came down with a brutal cold this week and did not accomplish as much as I would have liked to.  I did make some decent progress, however.

I spent an evening browsing the user documentation for the Myro library, which is a collection of Python functions to send basic commands to the Scribbler.  I also worked my way through “Learning Computing With Robots” by Deepak Kumar, a freely-available textbook about using Myro as a platform to learn Python.  In the process, I sharped my Python skills and learned my way around Myro.

My other, somewhat more significant advancement this week was successfully integrating Python, Myro and Prolog.  I found a hobbyist-made library called PySWIP to facilitate messaging between the Python shell and SWIPl, the implementation of Prolog that I am using.  Unfortunately getting this to work turned out to be a nightmare because of dependency and legacy issues… Myro runs best on Python 2.4, while PySWIP requires Python 2.7 and wouldn’t work with newer versions of SWIPL.  I had similar issues regarding to 32-bit version versus 64-bit versions.

I could not find any online to solve my quandary– I think I may be the first person to ever try using Prolog to control a Scribbler!  After brute forcing my way through several different combinations of versions, eventually I got Myro to run under Python 2.7 by updating all its dependencies manually, and found a version of SWIPL that was compatible with PySWIP.

So now I can import a Prolog file into Python by inputting:

> prolog.consult(‘sample.pl’)

And query it from Python by inputting:

> prolog.query(“male_ancestor(edward_vi, Ancestor)”)

Which returns a Python generator with all of the solutions that Prolog finds.  Pretty cool!