Records Missing From File After Writing

Dec 11, 2011 at 12:52 AM

I have an application now that writes STDF files real time. But I have a major problem.

In many cases the last records don't get written to the physical file on the hard drive. I see missing records like ptr, prr, wrr, pcr, hbr, and mrr.

Any idea why this is happening? and how can I fix this?

 

Thank you.

Coordinator
Dec 11, 2011 at 9:48 PM

Could you provide any code snippet? Mu guess would be that something isn't getting disposed properly so buffers aren't getting flushed to disk.

Dec 11, 2011 at 11:08 PM

I think you might be right, but there are some cases when we try to close the file and start another one before unloading the dll, and still we lose records.

But I think we figured a fix. We changed the StdfFileWriter constructor to create a stream with WriteThrough turned on as follows:

 public sealed class StdfFileWriter : IDisposable {

        RecordConverterFactory _Factory;
        Stream _Stream;
        Endian _Endian;
        public StdfFileWriter(string path, Endian endian) {
            _Stream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.Read, 1, FileOptions.WriteThrough);
            _Endian = endian;
            _Factory = new RecordConverterFactory(StdfFile._V4ConverterFactory);
}
 What I'm not sure about is the buffer size. If we are able to increase the buffer size safely then we can have more reasonable write time.

Dec 12, 2011 at 4:18 PM

So, I tried changing the buffer size from 1 to 512, and the issue occurred again. I started getting bad STDF files that are missing records. Some even missing the FAR record.

Coordinator
Dec 12, 2011 at 5:09 PM

Are you disposing of the StdfFileWriter when you are done writing to it?  I recommend using it in a "using" block so that it is automatically disposed when you're done "using" it.

Dec 12, 2011 at 7:21 PM
Edited Dec 12, 2011 at 7:24 PM

That's a good point. But what I have is a C# class library that receives discrete calls from the tester to write to the STDF file. So, each entry method writes a specific record or more to the file. Those methods use a static private StdfFileWriter object to access the file. So the file is really kept open as long as we are writing to it (I think). 

Now, I tried modifying LinqToStdf.StdfFileWriter to create the stream to Append. Then, I changed my class library to create a StdfFile Writer object local to each method and use "using" for that local object. But I got an InvalidOperationException for the second method called. I traced it; and I think it is because we lose the knowledge of Endianess when we create a second object trying to append records to an already existing file without writing FAR first.

Coordinator
Dec 12, 2011 at 11:09 PM

I think you're on the right track.  So, you can't use the writer in a using statement.  That's fine.  However, when you're "done" with the file, you do need to dispose of the writer (which will dispose the underlying stream, which will cause all buffers to flush), or you'll always have some variant of this problem.  So, your API needs to be expressive enough to capture "doneness".  Exposing your API as static without managing the lifecycle of the underlying file creates a scope mismatch between the lifecycle of the AppDomain and the lifecycle of the file itself.  Perhaps the operation that writes the MRR should trigger that disposal, and the operation that writes the FAR should trigger the initialization.

 

 

Dec 13, 2011 at 2:55 PM

Yep, that fixed it! All what I had to do is to call stdfFileWriter.Dispose() after writing the Mrr record.

Thank you very much for your prompt support.