Data Formats

The next generation software uses a self-describing file format designed specifically for SuperDARN. A brief description of the new format is given here. One of the long term aims of SuperDARN has always been to provide data in one of the "standard" formats used by the broader space physics community, such as CDF or netCDF. Incorporated in the new software are the tools to do just that.

Overview

One of the great advantages of self-describing formats is that it is possible to write a general purpose converter that translates one format to another - Any changes to the file format do not require the translator to be modified.

The tools included in the SuperDARN software convert from the SuperDARN internal format to netCDF or CDF, two of the most widely used formats. (HDF has a netCDF interface and works quite happily with netCDF files). Converting to these two formats also unlocks a host of other utilities and tools for working with the data.

The SuperDARN format was designed for storage efficiency and so does not include the extra meta-data components that are found in CDF and netCDF, however the software allows the user to add meta-data during the conversion process.

There are some problems with general purpose translators as they do not know of any implicit relationships between items in a data file. For example; SuperDARN data files are composed of records containing the radar operating parameters and observations. The time of each record is stored in the form of one variable each for year,month,day,hour,minute,second and usecond. When converting from the SuperDARN format to CDF, it would be nice to convert this time to a single variable witth the type "CDFEpoch", but a general purpose translator is incapable of doing this as it does not understand the relationship between the time variables.

Tutorial

The following is a brief overview and tutorial describing how the new file format data is converted to both netCDF and CDF.

dmapdump

The ".fitacf" and ".rawacf" files use the same self-describing file format called "dmap". (The illogical reason for this is that the library that works with these files generates a mapping from a data structures into a block of memory).

A useful utility is dmapdump which prints out on the console the contents of a dmap file. This allows you to inspect the contents of a data file:

>dmapdump 20021219.kap.fitacf
scalars:
	char	"radar.revision.major" = 1
	char	"radar.revision.minor" = 66
	short	"cp" = 9050
	short	"stid" = 3
	short	"time.yr" = 2002
	short	"time.mo" = 12
	short	"time.dy" = 19
	short	"time.hr" = 0
	short	"time.mt" = 0
	short	"time.sc" = 0
	short	"time.us" = 0
	short	"txpow" = 9000
	short	"nave" = 67
	short	"atten" = 0
	short	"lagfr" = 1200
	short	"smsep" = 300
	short	"ercod" = 0
	short	"stat.agc" = 8192
	short	"stat.lopwr" = 0
	float	"noise.search" = 6870
	float	"noise.mean" = 22870
	short	"channel" = 0
	short	"bmnum" = 15
	short	"scan" = 1
	short	"offset" = 15
	short	"rxrise" = 100
	short	"intt.sc" = 6
	short	"intt.us" = 0
	short	"txpl" = 300
	short	"mpinc" = 2400
	short	"mppul" = 7
	short	"mplgs" = 18
	short	"nrang" = 70
	short	"frang" = 180
	short	"rsep" = 45
	short	"xcf" = 0
	short	"tfreq" = 14565
	int	"mxpwr" = 4400000
	int	"lvmax" = 20000
	int	"fitacf.revision.major" = 5
	int	"fitacf.revision.minor" = 0
	string	"combf" = "$Id: normal_sound.c,v 1.94 2002/10/14 13:00:00 hughes Exp $"
	float	"noise.sky" = 6707.2
	float	"noise.lag0" = 0
	float	"noise.vel" = 0
arrays:
	short	"ptab" [7]
	short	"ltab" [2][18]
	float	"pwr0" [70]
	short	"slist" [46]
	short	"nlag" [46]
	char	"qflg" [46]
	char	"gflg" [46]
	float	"p_l" [46]
	float	"p_l_e" [46]

Note: By default dmapdump only displays the value of scalars but not the values in arrays (arrays are likely to be big), to output the complete file use the "-d" option: dmapdump -d 20021219.kap.fitacf

The remainder of this tutorial is split into two parts:

Converting to netCDF

Converting to CDF