Great progress in the SIE598 Data Stream Management class today. The assignment was to add the input of a ‘real’ stream to the STREAM manager.

One of the solutions was to write a program that converts a web-based stream (like Yahoo! Finance stock market updates, weather data XML files or data from a streaming simulation environment) into the STREAM .cvs format and push the new data into the data file associated with a stream schema in STREAM.

The challenge was to ‘slow’ down STREAM’s tuple read rate to be slower than the write (new tuple to file) rate. This allows to to actually compute streams instead of looping over a text file. Chris added a new input box interface to the STREAM GUI client to specify the tuple read rate exactly.

The second solution is to change the client file read code to work differently with eof markers in the file (which now either finish the execution or start an indefinite loop over the file) and add code to work with some kind of punctuation marker so that the processing either waits for new tuples, resumes partial processing or exits.

Advertisements