How to manage huge number of data (more than 18000 nulls)

Discussions about animating in SOFTIMAGE©
Post Reply
mattf06
Posts: 17
Joined: 22 Jan 2013, 11:25

How to manage huge number of data (more than 18000 nulls)

Post by mattf06 » 02 Jan 2014, 10:59

Hi all,

First of all, Happy new year.

I try to find out how to do this kind of video (http://422.com/work/422-south-data-visualisation-reel at 0:40).

I have data for a large number of aircrafts (from 18000 to 32000) over time. I'm able to import all of them in softimage as Null object, with animation path with a script from me. But, now I have to generate trails for each aircraft... I have look ICE Strands, but on 32000 aircrafts, I think is not possible to manage this amount of data.

Did anyone have a better solution to manage this kind of visualization?

thanks & regards,

Matt

Newcomer (<20 posts) alert: please use the URL tags - HB

Letterbox
Posts: 391
Joined: 17 Jun 2009, 14:49

Re: How to manage huge number of data (more than 18000 nulls)

Post by Letterbox » 02 Jan 2014, 12:09

mattf06 wrote:Hi all,

First of all, Happy new year.

I try to find out how to do this kind of video (http://422.com/work/422-south-data-visualisation-reel at 0:40).

I have data for a large number of aircrafts (from 18000 to 32000) over time. I'm able to import all of them in softimage as Null object, with animation path with a script from me. But, now I have to generate trails for each aircraft... I have look ICE Strands, but on 32000 aircrafts, I think is not possible to manage this amount of data.

Did anyone have a better solution to manage this kind of visualization?

thanks & regards,

Matt
Think smaller data sets, and compositors. If you look again at the video, how do you know that they didn't even render one strand line at a time and layer them in a compositor? Point is there is no way for the viewer to tell.

caledonian_tartan
Posts: 253
Joined: 17 Feb 2010, 15:13

Re: How to manage huge number of data (more than 18000 nulls)

Post by caledonian_tartan » 02 Jan 2014, 15:29

what if you put your nulls in a group, create a pointcloud and connect the group to a add point node and generate strand trails with that?
this shoud give you one strand trail for every null. 32000 particles is not very much. SI should be able to handle this, even with strands. otherwise you can cache overnight.
SI 2015 @ WIN7-64

Letterbox
Posts: 391
Joined: 17 Jun 2009, 14:49

Re: How to manage huge number of data (more than 18000 nulls)

Post by Letterbox » 02 Jan 2014, 16:48

Cal, thats a good point about putting the nulls in a group, but note grouping object can take some time - specially on large counts.

Thinking about this, i wouldn't use nulls paths etc, i'd try vectors - start and end with interpolate and do it all in ice, if you have a serious multicore machine, ICE is going to use all the cores, the null animate path might only use one core. And with that volume of objects, and unknown frame count, speed maybe an important factor. Just shows you Soft's capabilities, in solving this more than one way. Good luck.

NNois
Posts: 754
Joined: 09 Jun 2009, 20:33

Re: How to manage huge number of data (more than 18000 nulls)

Post by NNois » 02 Jan 2014, 17:26

hi, can you modify your script to build a point cloud file sequence ? this would be just perfect...

pezetko
Posts: 40
Joined: 09 Jun 2009, 01:58

Re: How to manage huge number of data (more than 18000 nulls)

Post by pezetko » 02 Jan 2014, 23:38

Try to build icecache directly from the data outside of XSI. E.g. with help of Bradle's Python ICE module it shoud be piece of cake. Look at this example http://www.cesarsaez.me/2013/11/scan-files.html

mattf06
Posts: 17
Joined: 22 Jan 2013, 11:25

Re: How to manage huge number of data (more than 18000 nulls)

Post by mattf06 » 03 Jan 2014, 09:21

thanks to all,

I will try icecache solution... :)

mattf06
Posts: 17
Joined: 22 Jan 2013, 11:25

Re: How to manage huge number of data (more than 18000 nulls)

Post by mattf06 » 06 Jan 2014, 10:31

hi all,

Ok, icecache is great because it is really fast. I can generate particles for each frame (5400 frames), but now, as I'm not ICE expert, how to generate strands for each particle?
do I have to create myself strands in icecache file? pretty hard, isn't it?

@caledonian_tartan
Put nulls into a group, why not.... in my previous test (generate 18000 nulls and assign a path animation) SI work pretty well.... except I'm not able to reload my scene !!! ????
So... is it possible to you to provide me a small example? Three animated nulls in a group and the ICE tree to generate strands from group?

thanks

Matt

caledonian_tartan
Posts: 253
Joined: 17 Feb 2010, 15:13

Re: How to manage huge number of data (more than 18000 nulls)

Post by caledonian_tartan » 06 Jan 2014, 11:27

...provide me a small example?...
shure. here you go.
btw. this example does not need any cache. it creates everything on the fly. although you can still cache it after that for faster feedback and avoid issues when reopening or rendering the scene.

NullsGroup2Strands.zip
(216.5 KiB) Downloaded 126 times
SI 2015 @ WIN7-64

mattf06
Posts: 17
Joined: 22 Jan 2013, 11:25

Re: How to manage huge number of data (more than 18000 nulls)

Post by mattf06 » 06 Jan 2014, 11:37

@caledonian_tartan
of course ;) so easy with ICE !!!
really thank you, you are my dude.

regards,

Matt

mattf06
Posts: 17
Joined: 22 Jan 2013, 11:25

Re: How to manage huge number of data (more than 18000 nulls)

Post by mattf06 » 13 Jan 2014, 09:24

@caledonian_tartan
I'm disappointed by multi-threading capability of ICE!
Handle 18000 nulls from a Group doesn't really use my 8 cores! 1 for sure, maybe 2 at least!

So, I don't get the right solution yet!

thanks to all.

User avatar
owei
Administrator
Posts: 840
Joined: 03 Jun 2009, 17:25
Location: Siegen/Germany
Contact:

Re: How to manage huge number of data (more than 18000 nulls)

Post by owei » 13 Jan 2014, 11:02

Hey!

You will have to consider, that the animation part in your setup, the animation of the nulls, will be single-threaded afaik. As ICE kicks in after the animation is done, this might explain why there is no full usage of all cores..
None the less, handling such amount of data is not easy and might take it´s time...on the other hand, why not dividing stuff into 3 parts of 6000 nulls each, caching the ice data for each chunk and joining only the ICE data into a new scene with 3 pointclouds?? Or even break it down in smaller chunks...once everything is cached, it should work pretty well...

cheers,
oli

mattf06
Posts: 17
Joined: 22 Jan 2013, 11:25

Re: How to manage huge number of data (more than 18000 nulls)

Post by mattf06 » 13 Jan 2014, 15:58

Hi again gurus....

Another idea... Maybe I can use my Nulls to generate point could cache file? But, is it possible to generate strands from point cloud cache file? Because, number of points (aircrafts) are not the same during time.

Just an idea... :D

regards,
Matt

Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests