What is this?
This is an android application that reads sensor data from the device and streams it to a Processing application that is running on your PC, controlling a cube on your screen with your phone’s/hand’s physical motion.
To establish the communication between the PC and the android in real time, it uses the oscP5 library for Processing that streams the data over OSC.
In one simple phrase: oscP5 with Android for Processing
The code and the apps
The code for the android application.
The code for the PC application.
The PC application for windows (.exe file).
The Android application (.apk file).
Remotely customizable 3D graphics
As part of a uni project I wanted to experiment in controlling 3d graphics on my PC screen using my Android phone.
The easiest way into interactive graphics applications is Processing. So, this is what I used to create the object to be controlled.
The next thing I needed was a way to stream sensor data from my phone to my pc and specifically to the 3d graphics application.
I thought of using one of the numerous Android apps available on Play Store that do just this, stream data from the phone to the PC. An example is Sensor Ex. However, I could not get any of these working, because they were assuming the two machines were connected to the same WiFi network, which is not always the case. I needed an application that would let you specify the target IP and preferably a port to which you would stream the data. I found an application that let you specify a custom IP, the android notifier, which pushed notifications from the phone to the PC, but the code wasn’t much help to me, it needed a lot of tweaking to implement sensor data streaming, let alone the fact that I would then need to create a separate script to stream the data from the “android notifier” desktop application into my Processing application and always need an intermediate to run it.
Luckily, there is a library (oscP5) that lets you send messages over OSC between Processing applications running on the same or different machines. I had come across this library about a year ago, during the preparation of an EEG-BCI workshop delivered by Panos Mavros and I to the UCL Coding Club. Panos had used the oscP5 library to stream data between Processing sketches for a simple EEG game we developed. Here you can find the overview of the workshop and here is the code for one of the games.
Furthermore, I had worked before with the Android mode of Processing and I knew about this great library (Ketai) that gives access to the android environment and all the sensors you could use.
So, the answer was right there in front of me, create two processing applications. One that will run on my PC and host all the customizable 3D graphics and one for my Android phone that will be reading the sensor data and streaming it to the PC app, customizing the graphics.
Creating the applications
The application creation was pretty straightforward.
First I used the oscP5broadcaster example to understand the way things work. oscP5 works just the same as fine on Android as on PC. The only thing that needed attention was the correct setting of the IP address of the PC, which must be the private one. Your computer’s private IP address will show up in the console if you run the “ObservedItem” processing sketch application. It will say something like:
This is you.
Another way to find your private IP address in Windows is by running the ipconfig command in a Command Prompt window. You’ll see your IP address in the IPv4 Address row beneath the name of your connection.
The IP address of your phone is not needed, since you will not be sending anything to the phone.
I created a start screen on the PC application that displays the IP of the PC so that the user can enter it on their phone without needing to search for it.
I used the java class InetAddress that can get the local host and its address:
I then used the Gestures and Multiple Sensors examples of the Ketai library, keeping all I needed and some more.
Here is the javadoc of oscP5 library that you might need if you want to change this application to match your needs.
What I did, was add to the (structured as an array) OSC message each sensor reading. Once the message is received, each array entry, accessed by get(), returns an OscArgument type object, which you need to convert to float, for the sensor readings.
Then, I only used 4 types of readings that control the rendering of a cube on your screen. The colour of the cube is determined by the light sensor, the size of the cube is determined by your zoom in-zoom out standard finger gesture, the rotation on the X-axis is determined by the rotation of your phone on the X-axis and the rotation on the Y-axis is determined by the standard finger rotation gesture on your phone’s touch screen.
Obviously, a lot more can be implemented and customized, this is just a showcase that demonstrates the potential. The successful connection of the android app and the processing app, the real-time data streaming and the 3d graphics physical reaction to physical motion. It feels as if you are holding the cube in your hand and shifting it to see it better.
The IP issue
The user has to input the private IP of the computer manually in the android application. Generally there is no way around the fact that the user will have to input manually one IP, either the computer’s or the phone’s. I tried the option of having the user input the phone IP on the PC and then the PC sending its IP to the phone, however I could not get my phone to receive any messages from the computer. I believe it was an IP issue, because I ran into 3 different IP’s supposed to be my phone’s. The actual private IP of my phone, found in Settings> About Phone> Status. The assigned IP by processing. The IP that the PC processing application logged it received the messages from. I tried all 3 IP’s, however I could not reach my phone. I don’t know what the real issue was. If someone has a better idea about it please write it in the comments. So, I decided to do it the other way round, type the IP of the PC on my phone. There is a small issue, if you do a mistake typing there is no way back and you have to restart the application, however it will be fixed in the future.
java.net.BindException: EADDRINUSE (Address already in use)
If you restart the Processing application a couple of times, there is the possibility that this error will show up in the console. Check that no other instance of the application is running in the background. Do make sure, because this is most probably (if not definitely) the reason for this error. It means that another application is already using this address, and most probably on your application is the one using it. I ran into this a number of times and I could not my head around what was wrong, since it was running just the previous moment. Go to task manager and check the processes and the number of java instances already running.