Data handling between VIs for idiots

[piw1uk]

Afternoon all,I am trying to find the most efficient way to pass data between VIs. I have a program monitoring an instrument and doing some processing of the data. In the background I have a VI logging diagnostics at 1Hz. Some of that data is averaged up from 10Hz data, but the displayed and logged are 1Hz. This diagnostics VI is crudely constructed with a massive While loop and wait functions. The main program needs the 1 Hz data at different points in its execution, certainly not every second. As I cannot pass data out of a While loop without stopping it, I am using global variables. I am self taught with Labview, and those around me who code in other languages frown upon the use of globals and I am trying to update my code and avoid if possible.Is there any way of efficiently transferring data from a While loop (or a timing structure if I get brave) other than globals?Paul

[F. Schubert]

You should use a queue. Search for Producer/Consumer Design Pattern. Felix

[Pnt]

Another option is a functional global.
How big is your data?

[Graziano]

Hi!    I suggest to begin with simple global variable, then you'll likely pass to Functional Global, which is a better option, but first just do some practice with globals.  They're quite straightforward to use.graziano

[piw1uk]

I am happy using globals, been programming for years (albeit basic stuff). They have caused me trouble when I've copied programs from one machine to another and I've used them to store calibration factors or when two programs have the same global variable. It is the method I've always used, I am being urged to move away from them.As for size, I reckon I will be generating an array of about 40 elements to save and display at 1 Hz for the diagnostics. The instrument runs from a few hours to several weeks. Some of the elements I will need to average, others I will just need the current value.Paulp.s. functional globals, time to search the help menu!Message Edited by piw1uk on 07-28-2008 09:45 AM

[Graziano]

Hi! &nbsp;&nbsp; they're not a built-in LV feature, you can build a functional global by hand... they're not strictly a LV component... look <a href="http://zone.ni.com/devzone/cda/tut/p/id/4338" target="_blank">here</a>! graziano

[Ben]

"p.s. functional globals, time to search the help menu!"
Hi Paul,
I wrote this <a href="http://forums.ni.com/ni/board/message?board.id=170&amp;message.id=240328#M240328" target="_blank">Nugget on Action Engines </a>just for this type of question.
I hope that helps,
Ben

[Mythilt]

A small warning about using Action Engines, globals and queues&nbsp;for passing of data between two seperately running VI's.&nbsp; If you use the programs as applications (ie, compiled into stand alone .exe's) they won't pass the information if they are compiled as two seperate .exe's.&nbsp; This is because AE's, Globals, and Queues are limited in scope to their parent Instance.&nbsp; While you are using the Development environment, even though the VI's are running seperately, they are sharing the same parent Instance (The LabVIEW developers environment.) but when you compile them into stand alone .exe's, they each will launch with thier own seperate parent instance (2 seperate RunTime environments) and thus won't pass between each other.
&nbsp;

[Ben]

Jon posted
&nbsp;




A small warning about using Action Engines, globals and queues&nbsp;for passing of data between two seperately running VI's.&nbsp; If you use the programs as applications (ie, compiled into stand alone .exe's) they won't pass the information if they are compiled as two seperate .exe's.&nbsp; This is because AE's, Globals, and Queues are limited in scope to their parent Instance.&nbsp; While you are using the Development environment, even though the VI's are running seperately, they are sharing the same parent Instance (The LabVIEW developers environment.) but when you compile them into stand alone .exe's, they each will launch with thier own seperate parent instance (2 seperate RunTime environments) and thus won't pass between each other.



For the most part I have to agree BUT...
In the case of the AE's it IS possible to opene an explicit reference to the AE in the application instance in which the AE is running and use VI Server call by refeernce nodes to use the AE. This technique can also be used between application instaces running on seperate machines.
Unless you are developing a plug-in architecture, I'd try to keep the app in one exe and make my life simpler. :smileywink:
BenMessage Edited by Ben on 07-28-2008 12:35 PM

[Kevin Price]

Functional Globals and/or Action Engines are good choices that have already been mentioned.
Queues were also mentioned, but as I understand your request, I don't think they'd be the best choice.&nbsp; It sounds like your "main program" will only occasionally need to consume a value, and I'm guessing that it'll want the most recent value.&nbsp; If so then Notifiers, which are programmed very similarly to Queues, would be a better choice.
An example of the kind of architecture I sometimes make: my data acq code pushes its data into a queue.&nbsp; I have a central data manager that pulls from this queue, perhaps does some processing and organizing of the data, and then pushes the results out into 1 or more queues and 1 notifier.&nbsp;&nbsp;Any file writing code will pull from a queue to be sure that data is stored losslessly.&nbsp; User display code and process monitoring code will often pull from the same notifier because they really only want on-demand recent updates.
One possible advantage of a Notifier to a functional global is the ability to wait for a new updated value.&nbsp; This can be very handy, depending on the app.
-Kevin P.

[Ben]

Kevin posted
&nbsp;
"
One possible advantage of a Notifier to a functional global is the ability to wait for a new updated value.&nbsp; This can be very handy, depending on the app.
"
Now if the AE fired off a Notifier when it executed.... :smileyvery-happy:
Thanks for that reply Kevin! It helped out with a design I am developing.
Ben

[piw1uk]

Thanks for all the replies, a few things to search for in the help menus.Kevin, you're right with your interpretation: the diagnostics are logged at 1 Hz for QA and any post poscessing and the main program uses only a small amount of it semi-randomly to generate the final producted.And I thought I was ok at LabVIEW programming! :smileyvery-happy: I guess there is more to it that reading in voltages and talking to serial ports!Paul

[DFGray]

I must disagree with a previous post.&nbsp; Queues are a great way to send information to multiple places, but only if you use the single-element version and treat them as a reference object.&nbsp; Otherwise, they are point-to-point (or many points to one point).&nbsp; In most cases, I prefer single-element queues over action engines/LabVIEW 2 globals due to their greater extensibility and better in-place performance.&nbsp; They also are more compatible with my preferred programming style (an accessor for each item in a reference object rather than a single object - action engine - to handle everything).&nbsp; They are easily embedded in LabVOOP objects to give you inheritance. Last year, I gave an NI-Week presentation on the various reference objects available in LabVIEW and their pluses and minuses.&nbsp; You can find a copy of it <a href="http://forums.ni.com/ni/board/message?board.id=170&amp;message.id=264865&amp;query.id=144585#M264865" target="_blank">here</a> - includes examples.&nbsp; The executive summary is simple - use single-element queues or LabVIEW 2 globals for most things; use globals for look-up tables.

[Pnt]

I have noticed that queues have a large performance hit, when you use the name to call an existing&nbsp;queue, instead of a wire.
This is the main reason i don't use them. Although i have not tried calling it by reference.....

[DFGray]

Acquiring a queue reference by name will give you a performance hit, although the option is nice to have.&nbsp; I always use them with a cached reference.&nbsp; There are several options for caching this reference: - Use a global.&nbsp; Globals are great for static look-up tables and this is a perfect use.&nbsp; Watch out for name collisions between different toolkits.
- Use a master queue to hold queue references to all your other queues.&nbsp; This is what I usually do.&nbsp; This results in never having to pass more than one queue reference into any subVI, so deep stacks of subVIs become manageable without huge numbers of inputs.
- Use a LV2 global.&nbsp; This is probably the best solution, since it is as fast as the global, and does not require a connection on the front panel.&nbsp; It is extensible, but once you get ten or so objects, it gets a bit unwieldy to maintain.

[Pnt]

DFGray wrote:
Use a master queue to hold queue references to all your other queues.&nbsp;


Nice trick ! Had never thought of it.

[Bob Y.]

Damien, Has this all been documented on NI's website somewhere other than just finding parts of it here and there in the discussion forums?&nbsp; I think that this would be a good topic for a white paper or even a web seminar but should at least be a topic for a knowledge base article. If one exists, could someone please post a link in this thread?Thanks,Bob Young

[Manc Pablo]

It seems to me like this topic has generated some interest and several options, I am ploughing through them at the moment. The stuff that DF Gray linked to has been very useful.The problem for a lowly programmer such as me has been moving away from global variables and wires to transfer data. If I use the same subVIs and instrument in a slight different configuration, traditionally I have saved a new application development in a different VI library to allow them to be transferred to other machines. Problems start to arise when these different applications are opened simultaneously and you have two global variables open with the same name in different locations. Only one can exist in memory.The dangers of being self taught. I also have to make it idiot proof so none labVIEW people do not fall into the same trap when simply running code.PaulMessage Edited by Manc Pablo on 07-30-2008 09:55 AM

[DFGray]

Manc Pablo, given that you copy code and reuse it (good for you!), you should probably use the single-element queue approach.&nbsp; Unless you specifically name your queues, copying code and reusing it will always result in unique data objects for each application.&nbsp; This is also very useful for multiple instances of the same application (but that is another topic).&nbsp; Both globals and action engines can get you into trouble with name collisions.&nbsp; A rename solves the issue in both cases, but it can be painful for a large application and the issue of what to rename to is always there.&nbsp; It also does not solve the multiple instance issue.Note that some of the rename issue can be avoided by putting things in libraries and LabVOOP objects.&nbsp; You then only have to rename the library or object, not every VI in it.&nbsp; You will still need to relink after the rename, though.Bob, I don't think this topic has been well documented by NI (one of the reasons I gave the presentation).&nbsp; I will see what I can do about that, since I wrote a paper for the presentation...

[Kevin Price]

DFGray,
I'm interested in your recommendation for single-element queues, and would like to know some of the practical implementation details.&nbsp;
1. Do you create the queue with size=1 originally?&nbsp; If so, does this allow LabVIEW to do some further optimizations?
2. How are "writes" handled exactly?&nbsp; I first picture doing a Deque (and discard) followed by an Enqueue of new data.&nbsp; Is this guaranteed to act "atomically"?
3. How do you choose reasonable timeout values where multiple processes may attempt simultaneous access to the single-element queue?&nbsp; Do you use infinite timeouts (-1)?&nbsp; Or do you use short timeouts and loop around your access attempts?&nbsp; Or some other scheme?
4. Suppose a process succeeds at Dequeue but fails to re-Enqueue leaving the queue empty.&nbsp; How do other processes handle that?&nbsp; Their Dequeues will keep timing out, right?
5. What about error handling?&nbsp; What are the scenarios where you need to add some smarts to the error handling?&nbsp; It seems to me that the possibility of scenario #4 above may dictate that one may sometimes need to perform an Enqueue even if the Dequeue produced an error.
6. When are they significantly more optimal than Notifiers?&nbsp;&nbsp; In the presentation, I see a Dequeue with data forked off to some processing code and also back to Enqueue.&nbsp; It seems that a data copy would be possible, maybe even likely for&nbsp;such a scenario.&nbsp; On the one hand, you want to re-Enqueue immediately to make the operation "atomic", but on the other hand this increases the likelihood that LabVIEW&nbsp;will need to make a copy of the data for processing.&nbsp; Right?&nbsp;
-Kevin P.,&nbsp; intrigued and curious

[Manc Pablo]

Wibble....:smileyvery-happy:Message Edited by Manc Pablo on 07-31-2008 10:08 AM

[tst]

I won't answer all the question for Damien, but I can answer some of them.



1. Do you create the queue with size=1 originally?&nbsp; If so, does this allow LabVIEW to do some further optimizations?


If you're diligent about the way you implement this (and I believe you have to be for it to work correctly), you don't actually have to set the queue size, but in any case, setting the size on a queue has no optimization value (at least according to the current documentation).



2. How are "writes" handled exactly?&nbsp; I first picture doing a Deque (and discard) followed by an Enqueue of new data.&nbsp; Is this guaranteed to act "atomically"?


That's exactly the concept, and it also has to do with your next question. The idea is that you always set the dequeue timeout to -1 and that you start every operation with a dequeue. That means that if the queue is empty, the entire read-write segment has to wait until the segment which currently has the element puts it back into the queue. This basically guarantees an atomic operation.
Regarding the data copies, I suppose that depends on the exact nature of the diagram. Remember that even if you want to re-enqueue quickly, LabVIEW probably does not "know" that you're using the queue to lock other sections of the code and will probably prefer to optimize the data copies. Of course, that's a guess, but not a wild one.

[DFGray]

A bit of code is worth a thousand words.&nbsp; I have attached the templates I use to create a standard reference object.&nbsp; These are written in LabVIEW 7.1.&nbsp; Use the In Place element to access the data elements if you have LabVIEW 8.5+.&nbsp; The code does not answer all your questions, however.- Setting the size does give you a bit of an optimization due to all element allocation occurring at queue initialization.&nbsp; However, in practice, this is meaningless if you always use only one element.&nbsp; I set the size to one element to make the object as robust as possible.&nbsp; I write infrastructure code that is used by programmers of many experience levels, so try to make things as easy as I can.&nbsp; It also helps keep me honest :smileyvery-happy:
- See the attached code for a write (I call it a "set").&nbsp; It is guaranteed to be atomic because nothing else can access the data since it is temporarily out of the queue.
- I have always used infinite timeouts for the queues.&nbsp; I can easily envision situations where you would not want to do that, but have never run into one.&nbsp; Note that the destroy VI does a force destroy, causing all get/set VIs which are waiting on data to immediately run with an error.&nbsp; If your program is designed well, either this will not happen or the program will expect the error and use it to take appropriate action (e.g. exit, abort, &amp;c.).
- If you fail to enqueue after a dequeue, it will cause a hang/timeout the next time an enqueue is requested.&nbsp; The easiest way to deal with this problem is to directly connect the error wire from the dequeue into the enqueue.&nbsp; Placing anything else between them will cause problems.&nbsp; You can combine errors at the end of the VI.&nbsp; I often isolate the error in/out of the enqueue/dequeue from the rest of the error flow until the end of the containing VI.
- If the dequeue produces an error, it did not dequeue data (you get a default data set).&nbsp; I can not think of any situation where you would want to enqueue data under that situation.&nbsp; A bad queue reference will simply produce another error.&nbsp; An input error to the dequeue means the data was not dequeued, so an enqueue will hang/timeout or cause a pending operation elsewhere to hang/timeout.&nbsp; A force quit will cause the dequeue to error, but there is no queue any more, so an enqueue is pointless.&nbsp; Run the error wire directly from the dequeue to the enqueue and it will automatically take care of most circumstances.&nbsp; Note you should only enqueue if the dequeue did not time out.
- Notifiers always make a copy for every reader and do not guarantee delivery.&nbsp; As implemented, with infinite timeouts, once the queue is empty, all other readers will wait until the enqueue occurs.&nbsp; No matter how long it takes, the process is atomic.&nbsp; Data copies are up to the programmer.&nbsp; Once the data is dequeued, it is possible to make many copies before passing one back to the enqueue.&nbsp; As mentioned above, the In Place element helps, but users of pre-8.5 LabVIEW can use the cluster "magic pattern" to reduce copies.&nbsp; Despite what the buffer viewer says, there really is no copy made when dequeueing the data.&nbsp; This is why the queue is such a good reference object.
Hopefully I answered your questions.&nbsp; If not, let me know...


VI Reference Class.zip:
http://forums.ni.com/attachments/ni/170/345908/1/VI Reference Class.zip

[tst]

DFGray wrote:
Setting the size does give you a bit of an optimization due to all element allocation occurring at queue initialization.

As you mentioned, it's meaningless in this case, but: "max queue size only limits the number of elements in the queue. It does not preallocate memory" (from the <a href="http://zone.ni.com/reference/en-XX/help/371361D-01/glang/create_queue/" target="_blank">LabVIEW 8.5 help</a>). It is important in queues which hold more data.

[Manc Pablo]

As you say, Mr DFGray, a bit of code speaks a 1000 words.I have generated a small simulation of what I need to do, which I have attached. Is this the sort of setup I should be using and the correct implementation of queues?And finally (I hope) I will be generating multiple elements. Is it better to: a) have multiple single element queues with different names? b) A single multi-element queue? or c) have a single queue with a single array of multiple elements? (assuming I have all the same data types)thanks in advance for yours (and everyone elses) advice.PaulMessage Edited by Manc Pablo on 08-05-2008 07:56 AM


data handling for idiots1.llb:
http://forums.ni.com/attachments/ni/170/346687/1/data handling for idiots1.llb

[Ravens Fan]

First, make sure you have a way of stopping your program besides using the Abort button.:smileywink:
&nbsp;
I wouldn't create one master queue.&nbsp; It would be putting all your eggs in one basket.&nbsp; And you don't want too many queues as that would be a lot to handle.
&nbsp;
I would recommend looking at all of your variables.&nbsp; Group them into ones that work together.&nbsp; Create a cluster that contains them and let that be the datatype for 1 queue.&nbsp; A group of related variables that are unrelated to the first would be a 2nd cluster and the element type for a 2nd queue.&nbsp; And so on.&nbsp; Use a handful of queues where the datatype is a cluster of related variables.

[Manc Pablo]

Thanks for the info. So what is the advantage of a cluster rather than an array?Paul

[Ravens Fan]

I can see 2 advantages.
&nbsp;
1.&nbsp; You can name each element of the cluster so that you can unbundle by name within your code and that makes the code self-documenting.&nbsp; With an array, you will have to remember or make notes as to what element 0 means, element 1,&nbsp; .....
&nbsp;
2.&nbsp; You can mix datatypes.&nbsp; For example, a boolean that indicates run status and a numeric that indicates speed could be put in the same cluster and remain bundled together.&nbsp; In an array, all elements have to be the same datatype, so you would have to translate the boolean to a numeric value such as 1 or 0 to store the data.
&nbsp;
One disadvantage is that you cannot scale you data structures at run time.&nbsp; For example, if you have a cluster of 5 different numerics representing speed, you can not change it while the code is running to add a 6th speed channel unless you've already accounted for that in your programming.&nbsp; But with an array, the user could define 1 to an infinite number of channels (within reason of course) during&nbsp;runtime&nbsp;and the program could iterate through the arrays to extract or replace values.
&nbsp;
There would be ways around this such as arrays of clusters, or clusters of arrays, or arrays of clusters of arrays,&nbsp; clusters, of arrays of clusters, .....&nbsp; and so on.&nbsp; Just a matter of which arrangement of structures gives you the flexibility you need for handling your data.Message Edited by Ravens Fan on 08-05-2008 11:44 AM

[DFGray]

I see Ravens Fan has given you all the info you need.&nbsp; I would like to add a couple of other points.&nbsp; When I start a LabVIEW project, I design it from an object-oriented perspective (a remnant of the days a decade ago when I programmed in C++).&nbsp; Each object is a collection of logically related collection of data and methods.&nbsp; The data goes into a strict typedef cluster which is stored in a single-element queue (I usually make the queue reference a strict typedef, too, as in the example code above).&nbsp; This determines how much data goes into each cluster.&nbsp; The methods are stored next to the data and namespaced with it.&nbsp; In LabVIEW 7.1.1 and before, namespacing was a prefix on the VI and control filenames.&nbsp; Now, namespacing is more easily done with libraries and objects.&nbsp; I would suggest using objects and making the object data your queue reference.&nbsp; That way you get inheritance too. If you are unfamiliar with object oriented programming, I would highly recommend you learn it.&nbsp; Like the shift from C++ to LabVIEW, there is a large context switch going from non-object to object oriented programming.&nbsp; However, it is worth the effort and will reward you with easier modularization and more reuseable code (but only if you actually stick with your pardigms:smileywink:).Let us know if you need more help.