fit and dat Files

For many years the primary archival data file types for SuperDARN have been the "fit" and "dat" files. The "dat" files store the raw ACF data derived from the radar observations, and the "fit" files store the fitted parameters generated by the FitACF algorithm. The formats for these file types were defined many years ago and require special software to read and write the binary format.

Although these formats have served SuperDARN well over the years, the limitations that they impose are starting to become a burden.

fitacf and rawacf files

The next generation of radar software will use a new self-describing format called "dmap" for storing both the raw ACF data and the fitted parameters. To distinguish the new format data from the old the suffixes "rawacf" and "fitacf" have been adopted.

Obviously there is a lot of legacy data in the old format and also a lot of legacy software that will only work with the old format. To solve this problem, tools have been developed for transforming the old data format into the new and vice versa.

dattorawacf and rawacftodat

To convert a dat file to a rawacf file, use the utility dattorawacf:

>dattorawacf 20021219.kap.dat > 20021219.kap.rawacf

The utility reads the dat file specified on the command line and writes the "rawacf" file to standard output (redirected to a file using ">").

The only command line flag is the "-vb" verbose option.

To convert a rawacf file to a dat file, use the utility rawacftodat:

>rawacftodat 20021219.kap.rawacf > 20021219.kap.dat

The utility reads the rawacf file specified on the command line and writes the dat file to standard output (redirected to a file using ">").

The -t command line flag sets a limit on the signal to noise ratio stored in the file. Ranges with a lag-zero power to noise ratio lower than this threshold are not stored in the "dat" file. The other command line flag, "-vb", sets the verbose mode.

fittofitacf and fitacftofit

To convert a fit file to a fitacf file, use the utility dattorawacf:

>fittofitacf 20021219.kap.fit > 20021219.kap.fitacf

The utility reads the fit file specified on the command line and writes the fitacf file to standard output (redirected to a file using ">").

The only command line flag is the "-vb" verbose option.

To convert a fitacf file to a fit file, use the utility fitacftofit:

>fitacftofit 20021219.kap.fitacf > 20021219.kap.fit

The utility reads the fitacf file specified on the command line and writes the fit file to standard output (redirected to a file using ">").

The only command line flag is the "-vb" verbose option.

dmapdump

The rawacf and fitacf files can be viewed using the utility dmapdump:

>dmapdump 20021219.kap.fitacf

By default dmapdump only prints out the values of scalar variables, to print out all data values you must specify the "-d" option.

Comparing dat and fit Files

One of the obvious issues when converting from one file format to another is to make sure that the transformation does not effect the values of the data in the file. The best way of checking this is to compare the differences between an existing file and a copy created by converting it first to the new format and then converting it back to the old. Unfortunately, due to the way fit and dat files are written it is impossible to do a byte by byte comparison. Instead, the two utilities cmpraw and cmpfit, can be used:

>cmpraw 20021219.kap.dat.old 20021219.kap.dat.new
>cmpfit 20021219.kap.fit.old 20021219.kap.fit.new

The utilities compare the fit and dat files specified on the command line and report any differences.