Current versions can read any files with several options to skip leading or
trailing lines and save data in columns of a DataFrame
. Numbers can be converted
into integers or floats. Furthermore, TUV 5.2 files (or versions with the same format)
can be read and j values saved to a DataFrame
together with solar zenith angles
in deg/rad in arrays all stored in an immutable struct TUVdata
.
The module is designed for Julia v0.7 and higher.
Install in the package manager with:
julia> ]
pkg> add https://github.com/pb866/filehandling.git
Import package with
using Pkg
Pkg.activate("path/to/Project.toml")
using filehandling
Omit the first 2 steps, if you installed filehandling
to the main julia environment.
To test the existence of a file use
ifile = filetest(ifile::AbstractString; dir::AbstractString="./")
where the directory of the file can either be specified as relative or absolute
path directly in the file name of ifile
or in a separate keyword argument dir
.
The function returns a string with the absolute file path or asks for user input, if ifile
doesn't exist. For non-existing files, an empty string is returned.
To read from a file use
content = readfile(ifile::String, headerskip=n, footerskip=m, dir::AbstractString="./")
where n
and m
are the number of lines to be excluded at the beginning
end of the file, respectively. The file directory is extracted directly from the
filename or, alternatively, taken from the keyword argument dir
as relative or absolute path.
To test the existence of a file use function filetest
.
More complex loading of data can be performed with loadfile
, where data is stored
in a DataFrame
with columns from the file data. Numbers and dates are automatically
converted to their respective types, and default error values can be specified for
missing or faulty data.
content = loadfile(ifile::String; kwargs)
The following keyword arguments are available:
dir
(String = "."
): Directory of the input fileifile
(can also be specified directly inifile
, the kwargdir
is ignored)x
(Union{Int64,Vector{Int64}} = 1
): Column index for column inifile
holding the x data (default column name in output DataFrame:x
orxi
, i = 1...n). Ifx
is set to0
, no x column is assigned and only y columns are used in the DataFrame (with default valuesy
oryi
, i = 1...n)).SF
(default value:1
(no scaling)): Optional scaling factor for all data.SFx
(default value:1
(no scaling)): Optional scaling factor for x data.SFy
(default value:1
(no scaling)): Optional scaling factor for y data.colfill
(String = "last"
): If the column length of the input file varies, the"first"
or"last"
columns of the file are filled witherr
(default values) according to the keyword. If you have a file with shorter columns to the right and the left, you either need to rearrange columns in the original data file or try to work with a specifically defined separatorsep
.ncols
(String = 0
): Defines the number of columns (x + y columns) in a file. If set to a0
, the number of columns is derived fromcolnames
or, if obsolete, from the first non-comment line of the file. You should only have to set the number of columns, if you have columns of different length with leading missing numbers and use whitespace as separator or if you want to exclude a large number of columns in your DataFrame.sep
(String
, default:whitespace
): You can specify any column separator with the keyword charactarsep
. Separators can be any unicode character (even special characters such as≠
orα
) or string series of unicode characters (including whitespace). The default splits using any number of whitespace, if you want to include empty columns, you need to specify the whitespace explicitly withsep
.escchar
(Union{Char,String,Vector{Char},Vector{String}} = ['\"', '\'']
): Characters or Strings used to escape Strings (separators are ignored within matching escape characters). A series of escape characters can be defined as vectors, but each sequence must be matched by the same escape characters, e.g."
and'
are used as default, but a sequence cannot be started with a single quote and ended with a double quote or vice versa.header
(Int64 = 0
): Optional line holding header names. Default values are used, if set to0
, positive values indicate the line number starting at the first data line (non-comment line or line afterheaderskip
), negative values indicate line number going upwards from the first data line (e.g.-1
is line above first data line). Routine may fail, if comment lines are used after skipped header lines.headerskip
(Union{Int64,String,Regex} = 0
): Define how many lines to ignore at the beginning of a file. If a string or regex expression is used, all lines up to the last instance are ignored.footerskip
(Union{Int64,String,Regex} = 0
): Define how many lines to ignore at the end of a file. If a string or regex expression is used, all lines starting from the first instance are ignored.comment
(String = "#"
): String that defines in-line and line commentserr
(Union{Int64,Float64,String,Missing,Vector{Any}}): Specify default values for missing data or data that cannot be converted to a data type (can include aNumber
,NaN
,missing
). By default,Int
andFloat
columns useNaN
and are always return asFloat64
,DateTime
usesDateTime(0)
. If no value could be converted in a column aString
vector is return, to allow text columns by default.coltypes
(Union{DataType,Vector{DataType}}
): If specified,loadfile
tries to convert each column into the specified type, on failure the default or specifiederr
values are used. Either useVector{DataType}
for each column orDataType
for a global value. (If errors are encountered in the data, 64 bit data types will automatically assigned.)colnames
(Vector{String} = String[]
): Specify header names for the output dataframe directly as kwarg. Overwrites values, derived from the input file.
To read the contents of TUV 5.2 output files use
jvals = readTUV(ifile::String; dir::String="./", DU::Number=350, MCMversion::Int64=4)
where ifile
is the name of the TUV output file with the absolute or relative folder
path given in the file name or the keyword argument dir
. DU
is the overlying
ozone column in Dobson units as defined in the TUV run. Data is stored in a immutable
struct
TUVdata
, where the fields jval
are assigned with a DataFrame
with the j values for every reaction using the TUV reaction labels as header,
solar zenith angles are stored in the fields deg
and rad
in the respective
units, the order
of magnitude of each jmax is stored in the
field order, the reaction labels are stored in the field rxn
, and the respective
MCM and TUV reaction numbers for rxn
are stored in fields mcm
and tuv
. To
assign the correct MCM reaction numbers, the MCMversion
needs to be and can be
inputted as:
2
: MCMv3.2 and older3
: MCMv3.3.14
: MCM/GECKO-A
The ozone column in Dobson units is stored in a field DU
as defined by the input parameter.
Function readTUV
returns a immutable struct TUVdata
with the following fields:
- jval::DataFrame
- order::Vector{Int64}
- deg::Vector{Float64}
- rad::Vector{Float64}
- rxn::Vector{String}
- mcm::Vector{Int64}
- tuv::Vector{Int64}
- O3col::Number
Function readTUV
works for every file in the format of TUV 5.2, however, if you
customised TUV and added reactions to the input files, you will need to amend the
database files in the data
folder. They are necessary to assign TUV and MCM
reaction numbers to the dataset.
MCM database files use a pipe (|
) as column separator, a hash sign as comment (#
)
and store the reaction number in the first column and the TUV reaction string
(from the I/O files) in the last column. Database files exist for 3 MCM versions:
- MCMv32.db for MCMv3.2 and older
- MCMv331.db for the most recent MCMv3.3.1
- MCM-GECKO-A.db for MCM/GECKO-A version, where the MCM is auto-generated with the help of GECKO-A
For the older MCMv3.2 files an additional column with scaling factors exist, which is ignored in this context.rxn number> | |
If you want to add/change reaction numbers, use the database files in the data folder
as template and add new numbers according to the above rules. Make sure, reaction numbers
are saved in the first and the TUV labels exactly as they are in the TUV input file
are saved in the last column of the database file. The first line of the file is reserved
for column headers, all other non-data lines need to be commented out with a #
-sign.
To obtain TUV numbers, a database file need to be saved to the data folder with the mechanism section from the TUV input file (see TUVrxns.db as template for the current TUV version).
If you add files or modify file names you need to adjust the file reading in function
get_photlabel
for the MCM database files and getTUVrxns
for the TUV database file
in TUV.jl.
- Revised exception handling: use
throw()
,@warn
, and@info
for errors/warnings
- Prefer package
import
overusing
- Revise kwarg
dir
in all functions to always be combined with the file name, even if the file name already holds folder paths. This allows the combination of@__DIR__
with relative folder paths scripts. - Add kwarg
dir
to functionreadTUV
- Bug fixes
- Revised database files for MCM and TUV reaction numbers
- Move data folder from src to main folder
- New kwarg
MCMversion
to specify MCM version number for correct reaction numbers - Change kwarg
O3col
toDU
- Use
readline
instead ofAtom.input
- Add kwarg
dir
for directory to functionreadfile
- Rename functions (
test_file
tofiletest
andread_data
toloadfile
) - Define escape characters to ignore separators
- Bug fixes
- Don't use separate splitting rules for column names to allow spaces in header names and avoid duplicates or unwanted results in column names
- Bug fix
- New Function
read_data
to load data from a text file into aDataFrame
and convert numbers and dates to their respective types (taken from module pyp: https://github.com/pb866/pyp.git) - Updated kwargs in function
readfile
: useheaderskip
/footerskip
forrmhead
/rmfoot
- Update
readfile
to Julia 1.0 standard
- Remove test print statements in
readTUV
- New fields
mcm
andtuv
inTUVdata
with vectors of integers with the reaction numbers for the labels inrxn
- Relies on the md file for reaction numbers in folder src/data/
- Field
order
in structTUVdata
is now a vector of integers
- Added field
O3col
of typeNumber
toTUVdata
to specify ozone column conditions in TUV runs with a new input parameter for functionreadTUV
(default:350
)
- Turn mutable struct
PhotData
into immutable structTUVdata
- Reduce fields to TUV data only without MCMparameterisations and statistical data
- New mutable struct
PhotData
containing TUV calculated j values, MCM photolysis parameters and related statistical data readTUV
now returnsPhotData
rather than a dictionary
- Strip reactions in DataFrame header with TUV data in function readTUV
readTUV
return dictionary with entries:jvals
,:deg
/:rad
(for solar zenith angles) instead of separate variables
- new function
test_file
to test the existence of a file
- Initial Julia package for simple file handling tasks with functions
readfile
to read numerical data from simple text files with certain formatsreadTUV
to read solar zenith angles and j values from TUV files