What is the format of Datalog files?

[wpd]

I would like to write an application to post process data collected in a Datalog file.  Can somebody direct me to the documentation that specifies the format of these files.  I can't seem to find it anywhere.  I have looked at http://digital.ni.com/public.nsf/3efedde4322fef19862567740067f3cc/747b83b6c8e4a6218625669100563b61?OpenDocument, but that does not match the format of the data I've got.  (The Datalog files I have all start with "DTLG" as the first 4 bytes and doesn't seem to have I32 bytes giving lengths of character strings and such.)Thanks for your help.--wpd

[Dennis Knutson]

"Datalog files" is a generic term. If you are talking about files created and written with the functions on the File I/O?Advanced File Functions>Datalog palette, then the format of the data is whatever is specified with the Open/Create/Replace Datalog function. In other words, anything you want. Do you have access to the code that created the files?

[altenbach]

If the application is also LabVIEW, things are easy. You simply need to know the record type and use tools from the datalog palette.
 
 
If the application is not LabVIEW, you probably should not be using datalog.
 
Quote from the <a href="http://zone.ni.com/reference/en-XX/help/371361D-01/lvconcepts/choosing_a_file_i_o_format/" target="_blank">LabVIEW help</a>:
&nbsp;
"Use datalog files to access and manipulate data only in LabVIEW and to store complex data structures quickly and easily"
&nbsp;
(I have not tried to reverse-engineer the format ;).)Message Edited by altenbach on 07-03-2008 11:51 AM

[wpd]

Hi,Thanks for the replies.The data was created within the File I/O&gt;Advanced File Functions&gt;Datalog palette, and I do have access to the code (in the form of walking over to the developer who wrote the code, tapping him on the sholder, and asking him if I can have access to the machine on which it resides).&nbsp; But I don't have access to a LabView license on every machine on which I might want to view this data.&nbsp; Nor do I have the patience to run through a GUI to extract and analyze the data.&nbsp; (That's what automated scripts are good at).&nbsp; Hence my desire to be able to post-process the data with custom code that I write myself.I don't want to reverse engineer the data format, I want to read and interpret the documented specifications for the format that is used.&nbsp; Thus far, I have not been able to find any such specifications, though I have read the same line you quoted about "Use datalog files to access and manipulate data only in LabVIEW and to store complex data structures quickly and easily".It will be quite disappointing to learn that such specifications are not available.&nbsp; I'll keep looking, and keep hoping that somebody will say "It's clearly documented in the LabView Data Formats manual", which I have yet to find.--wpd

[Dennis Knutson]

You don't understand. The specifications for the file is whatever the original programmer decided it should be. Since you have access to the source code, look at it if he did not document the file format somewhere. Creating a&nbsp;program to read the files is simply a matter of specifying the same data format that is in the write VI. If you are not familiar with LabVIEW, then post the VI that created the files.

[wpd]

You are entirely correct.&nbsp; I don't understand.The datalog file was created with the Open/Create/Replace Datalog vi.I know exactly what the Record Type was that was used to open the file.I know how he collected the data that was written to the file.I have the file on a separate PC.&nbsp; I know that the first 4 bytes of the file are the letters 'D', 'T', 'L', 'G', followed by binary data interspersed with some ASCII data like "Date/Time", "Waveform Name", "Average Count", etc... in other words, the names of the data paths that he bundled together.That header is followed by a bunch of binary data, presumably the timestamp at which he created the data, the name of the waveform, the average count, etc...What I don't know is the documented format of that file itself.&nbsp; From past experience, I would expect that there are records with some field (8, 16, 32 bits?) indicating a record type, record size, etc... followed by the data in big or possibly little endian format.From the research I've done so far, I'm not the first person to have asked this question, and I have yet to find the answer.&nbsp; I truly don't understand what NI gains by keeping it secret.&nbsp; So I'm hoping that the format of the file isn't secret, but just tremendously obscure.&nbsp; (Because I can figure out "tremendously obscure").Thanks again for the reply.&nbsp; It helps to talk/email about this.--wpd

[Dennis Knutson]

There is no secret that NI is keeping because they did not create the datalog files that you are trying to read. The format of the file is the record type that you mention in the Write File. Look at the shipping examples called Write Datalog File Example and Read Datalog File Example. In the Write example, the programmer decided to use a record type of a cluster with a string and a 1D SGL array. When the VI runs, that is the binary format of the file. In the read example, the record type from the writer was copied and specified. Then when the Read Datalog function is called, it returns a record that consists of cluster with a string and a 1D SGL array.
The programmer of the example could have just as easily chosen a different record type. He/she could have decided to save DBL data instead of SGL and that would change the file format and would require a change in the reader. And your original programmer is almost certainly using a different format. That is why I keep saying that the format of a datalog file cannot be documented by anyone other than the programmer. You have an infinite number of choices when using binary files because you not only have the various data types (I32, DBL, SGL, etc.), but you can define the data consists of clusters/arrays, etc. The original programmer could have also chosen to use the Write Binary function. It works like the Write Datalog but you also have the option to use big or little endian byte order.
I hope this helps. Take the record type from the write program and copy it into the shipping example. See if you get data that is meaningful. Otherwise, post the write program and a small sample file.

[wpd]

Hello Dennis,I've been out of the office for a couple of days and have now returned to this issue.&nbsp; Thanks for pointing me at Write Datalog File Example.vi.&nbsp; I have been studying it and have learned a lot.&nbsp; Unfortunately, I'm still at a loss to understand the format of the file produced by the example.&nbsp; I'll walk through what I've learned, and perhaps light will dawn on marble head (mine) as I do so.&nbsp; Or perhaps a light will shine for you that says, "Oh, I see what he's missing.&nbsp; If he just knew this blatently obvious piece of information, then he never would have had any questions to begin with.&nbsp; Let me tell him this and his confusion will vanish!"So... off to the races.I see that the example collects 10 records of 100 single precision random numbers.&nbsp; (The "10" and the "100" can be changed at runtime, but I'll go with these numbers for now).For each 100 sample record, a date and timestamp string is bundled with the sampled data.A datalog is opened (replacing an existing one with the same name, or creating a new one as needed) with the "Open/Create/Replace Datalog" Palette object.&nbsp; This object is provided a sample record type consisting of a bundle with an empty string and an empty single precision array.After opening the datalog, the example application enters a "while" loop in which it generates 100 (nominally) single precision random values (between 0 and 1) and a date/time string, bundles the string and the array of single precision values together, and hands that bundle to the "Write Datalog" Pallette object to scribble onto disk.&nbsp; It executes this while loop 10 times (nominally) and the application exits.That all makes sense to me.&nbsp; Now, I want to look at the file that was produced by running this example...The first 4 bytes of the file are:0x44, 0x54, 0x4c, 0x47, which just happen to be the ASCII characters 'D', 'T', 'L', G'.Those 4 bytes are followed by 41 various and miscellaneous binary valued bytes until the ASCII string "empty array" shows up in the binary data stream.Following that is a whole bunch more, mostly 0 bytes until finally, 582 bytes into the file, I see a 4 byte number (in big endian format) 0x00000013 (19 in decimal), an ASCII string reading "7/8/2008:3:29:44 PM", a 4 byte number (in big endian format) 0x00000064 (which coincidentally is equal to 100 in decimal), followed by 100 4 byte quantities that look suspiciously like they are single precision floating point values between 0 and 1.&nbsp; (They are probably in big endian order -- I haven't (yet) gone so far as to extract the binary values out of the file and attempt to convert them to floating point using big or little endian order.)&nbsp; BTW, the ASCII string "7/8/2008:3:29:44 PM" just happens to be 19 bytes long.This pattern: 0x00000013, a 19 character timestamp string, 0x00000064, 100 single precision floating point values, repeats 9 more times in the file.What I have done is to reverse-engineer enough of the file to be able to guess at its format.&nbsp; I knew going into the exercise that I was writing timestamp strings followed by 100 sample single precision arrays.What I am missing is the specification from NI that says:"This is the documented format of a datalog file.&nbsp; Every datalog file shall start with the 4 byte header 'D','T', 'L', 'G' followed by a 4 byte 'machine type' field indicating the type of processor used to create the data, followed by a 2 byte 'format' field indicating the format of the data, etc..." (I'm making that bit up).Right now, I have no expectation that data I write using another LabView application will start precisely 582 bytes into the file.Nor do I have any expectation that the data will always consist of a 4 byte length word followed by the data itself.Nor do I have any expectation that the data will always be in big endian format.I have a whole lot of

[Dennis Knutson]

Oh frack, I thought this was closed.
If you use the Open/Create/Replace&nbsp;Datalog function, you will always get the same header. The rest of the data can be anything and is defined in the program that does the writing. In the example, the data is a cluster that consists of a string and a 1D array of SGL. It could just as easily be just a 1D SGL array, a 1D DBL array, a cluster with several strings, or an infinite amount of other options. IT IS HOWEVER THE PROGRAMMER DECIDES WHAT DATA SHOULD BE STORED AND IN WHAT FORMAT. The record type you wire to the Open/Create/Replace Datalog file can be anything and this is where the format of the file is defined. You can find out how NI stores a SGL/DBL/I32,I16,array, cluster, but there is no generic definition of a datalog FILE. There never was and never will be. Please stop asking and looking for one. The way to find the definition of a binary file format is to look at the program that created it. Each program that creates a binary file can, and probably is, different in the way it defines the data that is being stored. It is the programmer's choice!
Also note that if the original programmer had decided to use the Write Binary and not the Write Datalog, the header would be completely different and the header format would be whatever the PROGRAMMER decided it should be.
I also keep saying that doing a read is simple if you have access to the write program. Look again at the read example. It simply uses the same record type from the write and everything after that is automatic. By using that defined data type, the Read Datalog function knows how to interpret the stored binary data.
Can we move on now? You are spending too much time on this and approaching it form the wrong end. If you had simply provided the code that was used to create the files and&nbsp;an example data file, the read VI could have been created in less time than I used to compose this response.

[wpd]

Hi Dennis, Thanks again for taking the time to reply.&nbsp; I think that it is possible that you misunderstood (or I forgot to describe) my original intent for asking about the format of the datalog file.I want to read a datalog file with code that I write outside of LabView.I can (and will have to) write my code so that it is aware of the format of the data generated by the LabView application.&nbsp; In the case of the Write Datalog File Example, my code is aware that the LabView application wrote bundles of data consisting of a date/timestamp string followed by an array of SGLs.You claim "If you use the Open/Create/Replace&nbsp;Datalog function, you will always get the same header."&nbsp; Where can I find the documentation that backs that statement up?&nbsp; I know that the header will change if I wired in a different record type, (even to the extent of changing the label attached to the "empty array" constant in the example -- that the header will change slightly).&nbsp; That's ok.&nbsp; I can live with that.You also said, "You can find out how NI stores a SGL/DBL/I32,I16,array, cluster ..." that is probably exactly what I'm looking for.&nbsp; Can you tell me where that is documented?I'm kinda stuck here.&nbsp; I don't believe that I've been able to articulate my question very well.&nbsp; I keep asking one question, and you keep answering a slightly different question.&nbsp; I am aware that the layout of a datalog file will change as a function of the type of data that are written to it.&nbsp; There are some specifications somewhere that govern how that layout changes.&nbsp; All I'm asking is, "where can I find those specifications?"&nbsp; I am not asking "what is the format of the datalog file generated by this or that particular application", I am asking "what is the recipe that describes the format as a function of the types of data written?"As an example, the "Read Datalog Example" is aware of that recipe.&nbsp; It knows that, to read a datalog with a record type of "a bundle consisting of an empty string followed by an empty array labeled 'empty array'", it must seek 582 bytes into the file to start reading it.Please don't answer with "THE PROGRAMMER DECIDES THE FORMAT".&nbsp; If you feel that that is the answer, then, once again, I have failed to explain the nature of my question and, once again, that answer won't satisfy me.Having said that, I truly apprecate the time and effort you expended to try to help me.Thanks again.--wpd

[Dennis Knutson]

For just information on particular data types and not information on 'datalogs' look at the Data&gt;Storage topic in the on-line help. On the header information created by the specific Open/Create/Replace Datalog, I don't know if or where this exists. I know the datalogs created from the front panel have changed from version to version and I don't think the use of the 'Datalog' functions are nearly as widespread as the ordinary Open/Create/Replace function that can create a binary file with or without a header and if a header is created, is entirely up to the programmer. In fact, I would recomend that the ordinary file read/write functions be used because even if you get a definition of the 'datalog' header, it is liable to change with a new release of LabVIEW.
And I guess I did not understand that you were not going to use LabVIEW to read these files. I think that is where the biggest misunderstanding was.Message Edited by Dennis Knutson on 07-09-2008 10:13 AM

[altenbach]

Maybe the original developer can make a DLL for you that can read that exact flavor of datalog files and output&nbsp;the desired data in some other format.

[wpd]

Now that's a clever idea.&nbsp; I hadn't realized that was a possibility.&nbsp; Could he (in the sense of "does the technology exist" not in the sense of "does he have the time and capability to") create a DLL which, when given a Datalog file, would produce a binary output file formatted according to my own specifications?&nbsp; Could I then link that DLL into whatever (and however many) applications I wish? I'll explore that option with him.&nbsp; In the mean time (for the next 15 minutes or so), I'll continue to see if I can find any documentation on the format of the file itself.Thanks for the idea.--wpd