
General disk access is one of the most bandwidth bound processes in this project where one transfers data from the disk to buffer to the CPU for calculations and then writing the results back to disk. The data is of course passed through levels of cache and memory which are faster at transferring data. This implies that disk access should be minimal. In this project the data to be filtered is larger than memory available, 8 GB, that it needs to be separated and transferred one block at a time. It is then important that the blocking is done such that one does not need to jump back and forth on the disk to read and write data, but rather that the data is aligned and is read sequentially, for optimal reads. 

In this implementation two files are used one to read from and another to write to. They both have the same layout and format such that one can use the result file to read as well if it is preferred. Both have the same header information in the same format as suggested and used by Schlumberger. After the header the data is presented in the form of a one dimensional line of floats. The data is to be read such that each row is read until the count of rows and then several rows are read until the count of columns, and this is to be repeated until the count of depth of the cube. This is how it can be represented as a triple for loop. But the data on the disk is stored aligned as a one dimensional array of float values.  

\begin{figure}[H]
	\centering
	\includegraphics[width=13cm]{report/method/blocking.PNG}
	\caption{Image illustrating blocking. (a) across x dimension, (b) across y dimension, and (c) across z dimension}
	\label{blocking}
\end{figure}

Since there are three dimensions, then one can block data across all three dimensions. To figure out which method is most useful, blocking is done across each dimension one dimension at a time and then tested to see the rate at which data is read and written (see Figure \ref{blocking}). This is also compared to reading and writing one float at a time just for comparison, and to illustrate how crucial alignment can be. The read and write methods in the developed framework only need to know what to read and the blocking parameters that are sent in as a pointer. While in the main method for-loops are used to iterate over the whole cube (See code segment below). 
\newline

\lstset{frame=single,tabsize=2, language=Java, basicstyle=\footnotesize, numberstyle=\footnotesize, numbers=left, captionpos=b, breaklines=true, title={Code in C for reading a block of data from bin file}}
\begin{lstlisting}
void readCube(FILE *file){
	_fseeki64(file, fileOffset, SEEK_SET);
	data = (float*)malloc(blockLength * sizeof(float));		
	fread(data, sizeof(float), blockLength, file);
}
\end{lstlisting}


In both C and C\# versions of the code, predefined functions are used to read and write to files on the disk. These methods need to know the path of the file to be read, a pointer in memory to where the data is to be read,  a mode in which to open the file such as binary, and the amount of data to be read. The read method used can be found on line 4. To set an offset as to where to read and write in the file, one can use another built in function to seek in a file. This can be found on line 2. Notice that for large data set the conventioal seek function cannot be used because it does not address 64 bit of memory. The use of the offset is important when blocking such that one can set the pointer to where one wants to continue reading from disk as the next block is about to be read. For more documentation on these functions see Microsoft MSDN \cite{micro_msdn}. Similar functions where used in the C\# version of this code. In both cases the data is read into an array in memory that will further be used in the convolution algorithm.

As mentioned before when blocking the data one needs to consider the shared data between blocks, which grows as the filter size grows. But, is overall of little significance since there is such a vast amount of data to be processed. To include this data in the borders one can just adjust the offset sent into the read method to read earlier and stop later in all dimensions depending on the size of the filter.  We will have a closer look at this later when discussing the convolution algorithm.


