Memory/Speed of Split 1D array vs Delete from array

[DavidU]

Just wondering how Split 1D array works - does it create two new arrays in a new section of memory or does it just split the existing array in two, reusing the memory in place. (assuming that the original array is not needed elsewhere). If the latter is the case then presumably it is more efficient to use split array than delete from array if I want to remove element(s) from the beginning or end of the array. Is there a speed advantage as well? If I use split array but don't then use one of the output arrays is that memory deallocated or does the array remain in memory?ThanksDave

[muks]

Well I tried the profile Vis to check If there is any difference with respect the execution time of both.I didnt find any difference.

[tst]

I would suggest simply testing it with a large array.
Based on experience with the way these things work in LabVIEW, I wouldn't be surprised if both cases were optimized not to cause data copies. If memory serves (I don't feel like looking it up and I may very well be wrong), an array has the dimension sizes together and then a pointer to the actual data. If that's true, then there shouldn't be a problem splitting it (or deleting either from the start or the end) in-place. You should try looking up the documentation for how LV holds arrays in memory.
This of course might be different for arrays where the elements have variable sizes (e.g. strings, variants, pink clusters).
If you do this research, post back to tell us the results. It might be interesting.
P.S. I seem to remember something about a bug in an older version which caused LabVIEW to leak memory if the bottom output of the Delete primitive was unwired, although I don't remember the specifics. Presumably, this was fixed, and if you don't wire the array out its memory should be freed (within the general memory handling behavior of LabVIEW, of course. If you want to make sure, use the Request Deallocation primitive).

[altenbach]

It seems to me you want to obtain an array subset (since "split array" and "delete from array" can be used for what you want).
 
In this case you should use "array subset". It is the correct tool for the job. :)
 
(still I recommend to do some beckmarks with typical data. Is this a one-time operation or do you do it repeately in a loop, for example?)Message Edited by altenbach on 08-11-2008 12:30 PM

[Wiebe@CARYA]

If you are interrested in the first part of the array, you can also use the
Resize Array function. I think split string is actually more efficient, but
haven't tested this for a long time. It might also differ between LV
versions.

Regards,

Wiebe.

[DavidU]

Hi,Thanks for all the replies. At the moment my array is very
small, and it probably doesn't make any difference which way I do it. I
asked more out of interest than anything else. I'll follow up some of
your reading suggestions.I have done some benchmarking for the
different methods available on a 1D Double array, results below. Each
value is an average of 100 trials. Times in ms.Interestingly, 
it depends on whereabouts you divide the array, particularly for Subset
and Split methods, less so for Reshape . In the table below the (split
point)*(array size) gives you the index where the array was divided.Also
note that the choice of method will depend on what you want out at the
end, Subset and Reshape only give you one array out, whereas Delete and
Split give you two arrays.ThanksDave<img src="Loading Image..."> Message Edited by DavidU on 08-12-2008 12:24 PMMessage Edited by DavidU on 08-12-2008 12:25 PMMessage Edited by DavidU on 08-12-2008 12:27 PMMessage Edited by DavidU on 08-12-2008 12:27 PM


Benchmarks1.png:
http://forums.ni.com/attachments/ni/170/348554/2/Benchmarks1.png

[tst]

It looks like your system is quite unreliable. :smileyvery-happy:
I would suggest getting rid of the smaller arrays. They don't give enough accuracy anyway.
Also, I would advise wiring each of the outputs&nbsp;into an index array and wiring an indicator out of that to have&nbsp;consistent memory allocation.
Additionally, either closing LabVIEW or allocating a large amount of memory for LabVIEW in advance might help.
If you post the VIs, other people can test this too. :smileywink:

[DavidU]

Ok please ignore the results I posted earlier, they are rubbish - not least because I got the column headings mixed up:smileymad: but also because my code was full of bugs :smileysad:So, here is a revised table, and the code - which hopefully only contains a few bugs... I'm not clued into benchmarking yet so please feel free to rip the code apart.I still get different results depending on where in the array I split it, most noticeably with subset and reshape. There is no effect with split. I'm guessing this is to do with the memory allocation. (I have not preallocated memory in my code, but I did wire the output arrays to Index Array)<img src="Loading Image..."> Message Edited by DavidU on 08-12-2008 04:49 PM


Benchmarks 2.png:
http://forums.ni.com/attachments/ni/170/348680/1/Benchmarks 2.png


split array test.vi:
http://forums.ni.com/attachments/ni/170/348680/2/split array test.vi

[altenbach]

As expected, "array subset" is the correct way to do things and the time is somewhat proportional to the size of the resulting array, again as expected.
&nbsp;
About your benchmark code, a few things could be improved:

- You should disable debugging.

- To compare identical outputs, you should only retain one relevant array from the case structure, otherwise you are comparing apples and oranges.

- You should NOT place indicators inside a frame that you are timing, else you run the chance that sometimes you measure&nbsp;the updating of&nbsp;the indicators. Even then, asynchronous updated of the FP indicators could randomly interfere with the timings, especially&nbsp;on single processor systems. You should use a three-frame flat sequence for the tick-code-tick and it should not contain any controls or indicators or possible parallel code.

- I would increase the priority and lower the loop count.

- Taking the "mean" is mathematically incorrect. You should use "array min". All external influences add to the "real"&nbsp;execution time!

[tst]

I'm not sure that your testing is accurate, because you might have memory allocation issues (going over the same array in the loop might force LabVIEW to create a new copy of the data each time. Unlike Altenbach, I was expecting both split and and delete to reuse the memory if possible, so I wrote a different implementation (8.0).
&nbsp;
I'm not actually sure if this implementation is any more accurate, but at least for me subset, reshape and split all take 0 ms. That makes sense since you should be dealing with pointers (if you look at the context help for split you will see it returns subarrays).
&nbsp;
Delete, which I was expecting to be optimized the same way, does seem to create a copy. I'm not sure why that is, since it should be able to behave the same way that split does, at least for the case where the array is not used elsewhere.


Test Times.llb:
http://forums.ni.com/attachments/ni/170/348737/1/Test Times.llb

[tst]

OK, I made a stupid mistake in the last post. I removed the case structure, which removed the timing dependency. I used the 8.6 auto cleanup tool, so I didn't even notice it.
&nbsp;
Here's another version (8.0), this time hopefully more accurate. In this version, both reshape and delete take some time. The others take no time.
&nbsp;
Again, I would expect that those two at least could have optimizations for a case like this, but I guess they don't.


Test Times.llb:
http://forums.ni.com/attachments/ni/170/348884/1/Test Times.llb

[DavidU]

so I've redone my test code and am getting similar results to yours. Now&nbsp; I just have to make sure my test code is right!
&nbsp;Thanks for all the helpDave

[Ben]

Yair,&nbsp;You are running all of the benchmarks at the same time rather than one after the other. I believe the fighting of the processor will screw up the numbers while more than one is running. Once three have finished the slowest (and still runnning) will then get exclusive access of the CPU and will start to run faster. I will generally make sure the individual test run one afte the other.&nbsp;This image (from the clear as mud thread) illustrates how I do most of my bench-marking.<img src="Loading Image..." border="0"> &nbsp;Ben

[tst]

You're right. My modification was only a quick one to demonstrate why using a loop is a problem here. That was also the reason for my original mistake.&nbsp;Ideally, the arrays should be allocated first and then each of the subVIs should be tested separately.