Any interest in Kinect for Windows & XBox gamepad drivers?

[Eric Cosky]

Hi,

I recently wrote an application that captures bone data from a Kinect for Windows devices (Important: this is not the XBox Kinect) and sends it to a device driver running in Softimage. It is a functional solution for getting the raw skeletal motion data into Softimage using the device driver interface. I know there is already an ICE-based Kinect solution out there, but this takes a different approach by using a device driver which I find a little more to my liking than using an ICE node. I can observe values in a custom parameter set and pull the values from there to do whatever I need with them such as binding in ICE or more basic expressions. Plus I don’t have to run the simulation to see what is going on, you just activate the driver and it works even while Softimage is sitting in pause mode.

I also wrote a driver that is designed to work with Xbox-style controllers. The input data is provided as positional data and all driver item names are mapped to Xbox controller names.

I’m wondering if there would be any interest in a bundle of these two drivers & the support application for something around $50. I am really under the gun to get my next game out (I’m a solo indie game dev) and I just can’t spend any more time on it unless I have some hope it can pay for the time it would take me to make a proper release out of it.

I don’t want to assume anyone has any interest at all, and won’t be surprised if there isn’t. I wrote these for my own needs and find them useful so I am happy with that. However, if these sound like tools you would like, please let me know. If enough people want access to them then I can wrap up a few loose ends, tidy up the UI, do some more testing and take care of various other things necessary for it to be ready for general use. If a lot of people expressed interest I would be keen to add support for Kinect face tracking data as well as a plug-in mechanism to allow people to easily filter the skeleton data with a .NET DLL prior to handing the data to Softimage. It’s all doable but I can only take the time if it makes business sense which I can only hope everyone understands.

Here are some screenshots:




Please let me know if you are interested and we'll see what happens, thanks.

[caledonian_tartan]

i'm fundamentally interested!

do you have a demo video to see it in action?

[Eric Cosky]

I just made a very quick capture of running Softimage with the kinect driver capturing a live stream, http://youtu.be/cc66W0c5vHA

As I said above I can't spend much time on it yet so I hope this gives you a sense of what it can do in practice. It's a bit noisy in this video- I made no effort to get a clean capture, this is just what it recorded for me after I read your post a few minutes ago.

[dwigfor]

I might be interested in this. How well can it record/plot the motion capture?

I wonder if it would be possible to see the video stream/setup in a CustomDisplayHost window. Seems like that's one of the least utilized features of Softimage... I dunno how hard that'd be to implement, but I thought it might be possible.

[Eric Cosky]

It seems to record the data correctly, to the best of my knowledge and ability to test. Like anything moving data around, there is a bit of latency that varies based on how many UIs are visible and being updated as well as scene complexity, but it seems to accumulate the data and apply it at the correct time rather than skip or drop data.

The companion app has a view of the video & depth stream coming from the Kinect which does a good job of letting you see what the source data is doing before it gets into Softimage.

[Nick3d]

I'm quite interested too, I'm about to get a Kinect to work with ipi Desktop Motion Capture, but instead of having another app to process the motion capture I rather do everything inside Softimage.
There's a guy who has built more or less a similar system based on Kinect hooked up into SI, its called Kinemocap and it's 99€
If you fix the "noise" problem I'm willing to pay 50$ easily.

Another thing: it would be possible to do a similar system for facial motion capture? I explain myself a little bit better:

Instead of tracking bones you track the dots on the face based on the colour, with Kinect ( don't know if you're using it ) you get the depth information so you have a 3D movement instead of a 2d movement....but even just 2d movement would be good as well...then you hook up ( parent ) everything into Facerobot or your rig...
Even better to stream Kinect video into Softimage on a grid ( probably I'm talking BS here, but I'm guessing... ) and get the marker position of the face via Kinect in realtime ( Paul Smith did a tracker based on markers that works perfectly based on ICE ) and hook up everything into Facerobot or whatever...

For both scripts I'll easily give you 100-120$
Good luck with your game, I'm developing too, and since I'm not so good in animation I'm trying to see the best solution out there, this could be a really good one

[Eric Cosky]

I was very close to committing to this project but after much thought and discussion I decided I need to stay focused on finishing up the game I am working on right now. The game has been over a year in the works and despite the appeal of all this, I really can't let myself switch gears to work on a another project right now - I just have too much invested in the game and I need to get it done. As much as I'd like to be able to make a public release of the mocap tools, I know it wouldn't get my full attention and I don't want to do something unless I can take the time to really do it right and make it something of useful value and I just can't do that right now. Hopefully I can revisit this project in the future.
Thanks

[Kerro Perro]

Well good luck on your game Eric, i can only respect your decision and am curious to see what one man can make game-wise, you got a blog or something?

And just to answer Nick3d there is a Kinect facial mocap system out there: http://www.faceshift.com/ I got the beta but totally havent found time/motivation to try it out yet....

[Nick3d]

I was very close to committing to this project but after much thought and discussion I decided I need to stay focused on finishing up the game I am working on right now. The game has been over a year in the works and despite the appeal of all this, I really can't let myself switch gears to work on a another project right now - I just have too much invested in the game and I need to get it done. As much as I'd like to be able to make a public release of the mocap tools, I know it wouldn't get my full attention and I don't want to do something unless I can take the time to really do it right and make it something of useful value and I just can't do that right now. Hopefully I can revisit this project in the future.
Thanks

I know what you mean, game developing takes time...lots of time, so I can understand perfectly why you don't want to lose time in other stuff...
Which game engine you're using? If you're developing with Unity or UDK I did some videotutorials on how to export animated characters from SI to those engines, if you're interested check them out
http://www.youtube.com/user/Nick3d/videos

@Kerro Perro: I just received the beta yesterday and I'm going to try it as soon as I'll buy the Kinect...
But, you know, it would be nice to have everything inside a single 3d app instead of exporting/importing, at least you don't spend much time learning new tools and usually there are always some problems with export/import...but I might be kinda lazy lately
Thanks anyway

[Kerro Perro]

Sure, np.

I actually bought a kinect last year for mocap as well but frankly i've been mostly just playing with it ;)

I did do some tests: first with Brekel Kinect, it might be better now but at the time it's tracking was awful and it required two people to run so no go. Second i tried iPi and that did work a lot better, first of you record your footage fisrt and then edit and convert plus it's tracking was a lot better. Then you just output as fbx or bhv and import that in SI as any other piece of mocap. If you want to see what that's like you should get this huuuuge free collection of mocap: https://sites.google.com/a/cgspeed.com/ ... on-capture They are in bhv so i actually convert them to fbx in MoBu because the Exocortex rig (wich i see you use as well) responds better to that.

And yeah i know all those different apps are a hassle but IMO a few extra steps to get good clean animation is worth it. I wish it could ALL be done in SI including sculpting and editing but i think that will always be a nice dream, plus there isn't much to learn about iPi and my knowledge of MoBu is just enough for conversion, no more ;)

[Nick3d]

Sure, np.

I actually bought a kinect last year for mocap as well but frankly i've been mostly just playing with it ;)

I did do some tests: first with Brekel Kinect, it might be better now but at the time it's tracking was awful and it required two people to run so no go. Second i tried iPi and that did work a lot better, first of you record your footage fisrt and then edit and convert plus it's tracking was a lot better. Then you just output as fbx or bhv and import that in SI as any other piece of mocap. If you want to see what that's like you should get this huuuuge free collection of mocap: https://sites.google.com/a/cgspeed.com/ ... on-capture They are in bhv so i actually convert them to fbx in MoBu because the Exocortex rig (wich i see you use as well) responds better to that.

And yeah i know all those different apps are a hassle but IMO a few extra steps to get good clean animation is worth it. I wish it could ALL be done in SI including sculpting and editing but i think that will always be a nice dream, plus there isn't much to learn about iPi and my knowledge of MoBu is just enough for conversion, no more ;)

Thank you very much for the link, very helpfull
I'm going to try with ipi and Kinect...I did some tests with 3 cameras but its time consuming a bit too much, so I'm going to Kinect

[Eric Cosky]

@Nick3d, I wrote my own engine - part of the reason why it's taking so long! A big part of the motivation for it was I was very frustrated with the process of getting animations from Softimage into a game. I wrote an exporter that packages the scene, including all meshes, instances, textures, shaders and actions into something that loads directly into an immediately usable form in the engine without all those import hassles. The only thing that needs to be manually configured at this point is the per-character animation tree logic. Hopefully it will have been worth the investment, however I do look over with a tinge of jealousy at the very cool things Unity & UDK can do and wonder from time to time if it would make more sense to shift over to an off the shelf solution. Who knows what the future holds.. I really do like having ownership of the entire solution though. Nice to know exactly what is going on.

FWIW, I also have an iPi setup. Its functional, but (for me) frustrating to use. I don't care for the UI. I'm also not really a fan of the price/value ratio for upgrades to the current version - many of the features in the upgrade I think should have been provided as a free update. So for me, I'm mildly disappointed with iPi but perhaps I am just not a good fit for their products. There seem to be a lot of happy users out there and it is arguably hard to beat right now. Regardless, I've decided to just get by with the old version and my own tools and save up for something else at some (probably distant) point in the future.

[Nick3d]

@Nick3d, I wrote my own engine - part of the reason why it's taking so long! A big part of the motivation for it was I was very frustrated with the process of getting animations from Softimage into a game. I wrote an exporter that packages the scene, including all meshes, instances, textures, shaders and actions into something that loads directly into an immediately usable form in the engine without all those import hassles. The only thing that needs to be manually configured at this point is the per-character animation tree logic. Hopefully it will have been worth the investment, however I do look over with a tinge of jealousy at the very cool things Unity & UDK can do and wonder from time to time if it would make more sense to shift over to an off the shelf solution. Who knows what the future holds.. I really do like having ownership of the entire solution though. Nice to know exactly what is going on.

FWIW, I also have an iPi setup. Its functional, but (for me) frustrating to use. I don't care for the UI. I'm also not really a fan of the price/value ratio for upgrades to the current version - many of the features in the upgrade I think should have been provided as a free update. So for me, I'm mildly disappointed with iPi but perhaps I am just not a good fit for their products. There seem to be a lot of happy users out there and it is arguably hard to beat right now. Regardless, I've decided to just get by with the old version and my own tools and save up for something else at some (probably distant) point in the future.

Wow, I really admire your commitment to the job, you also wrote your own engine, that is something, let me tell you!
I also would like something where you have control of basically everything, but programming for me was always something not interesting, so I never tought that lately it would be helpfull...
Anyway I really hope the best for your project, please keep us updated ( or blog, twitter, youtube, whatever ) because I'm really curious about it!

[Eric Cosky]

Thanks, Nick

I haven't made many public updates in a while, trying to stay focused on the game dev itself until I reach some kind of alpha but here is a video I made back in January (wow has it been that long!?!?!) http://youtu.be/T48eNKBJI8M



Also, here is my blog link: http://blog.boundingboxgames.com/
I do intend to resurrect the blog & youtube videos once the game approaches alpha which will hopefully be very soon, this can't go on forever!

Cheers,
Eric Cosky
https://twitter.com/ericcosky

Moderator edit: our youtube tags only accept the ID of the video, not the full link - HB

[Nick3d]

Thanks, Nick

I haven't made many public updates in a while, trying to stay focused on the game dev itself until I reach some kind of alpha but here is a video I made back in January (wow has it been that long!?!?!) http://youtu.be/T48eNKBJI8M



Also, here is my blog link: http://blog.boundingboxgames.com/
I do intend to resurrect the blog & youtube videos once the game approaches alpha which will hopefully be very soon, this can't go on forever!

Cheers,
Eric Cosky
https://twitter.com/ericcosky

Pretty cool!
Keep working on it ;)

[sirdavid32]

How has it been since you started your project? I mean: with 2015 SI being EOL, what are your plans for the controllers?
Do you plan to make new versions?