Working with:
__ ___ __ _ \ \ / / |/ /__ _(_)___ ___ _ __ \ \ / /| ' // _` | / __|/ _ \ '__| \ V / | . \ (_| | \__ \ __/ | \_/ |_|\_\__,_|_|___/\___|_|
Golang package for reading and writing FoxPro dBase tables (DBF), databases (DBC) and related memo (FPT) files.
License: BSD 3-Clause "New" or "Revised" License
A code page is a set of characters specific to a language or hardware platform. Accented characters are not represented by the same values across platforms and code pages. In addition, some characters available in one code page are not available in another.
https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/8t45x02s(v=vs.71)
When new entries are written, the NullFlag column is not written. So that there are no variables and zero fields in written entries.
Visual FoxPro uses tables to store data that defines different file types. The following list includes the file types that are saved as tables:
- Database (.dbc)
- Form (.scx)
- Label (.lbx)
- Menu (.mnx)
- Project (.pjx)
- Report (.frx)
- Table (.dbf)
- Visual class library (.vcx)
You can use and browse these files in the same way that you browse any table file because these files are actually tables.
https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)
Currently, tables can be read and written, but for this the files must always already exist. It would be nice if you could also create new files with the package.
I'm not sure if there is an issue with my Go environment on macOS, but when I try to use the module (for example by running the examples), I get syscall.Handle
errors:
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:13:27: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:14:27: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:121:28: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:149:31: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:151:36: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:199:32: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/table.go:76:40: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/table.go:88:32: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/interpreter.go:98:33: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/interpreter.go:104:36: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:151:36: too many errors
Is there a known issue with running this module on macOS? (M1, Ventura 13.1)
The "TrimSpaces" option in the dbase.Config is only used in the ToStruct -> ToMap functions
When getting the values directly the spaces remain in the result
Describe the bug
On windows my file isn't recognized and throwing dbase-io-windows-findfile-1
. My code uses filepath.Join(pi.Config.DBPath, 'daten', 'my-table.DBF')
which works on my linux clients. However on windows the path with the config prefix does not work. I printed the path Y:\some-folder\daten\my-table.DBF
already and checked if it exists by entering it to win+r
and everything worked as expected. I am a bit lost where the issue could come from.
To Reproduce
Example code to reproduce the behavior:
package main
import (
"fmt"
"github.com/Valentin-Kaiser/go-dbase"
)
func main() {
configPath := 'Y:\somefolder'
p := filepath.Join(configPath, 'daten', 'mytable.DBF')
table, err := dbase.OpenTable(&dbase.Config{
Filename: p,
TrimSpaces: true,
})
if err != nil {
return dbase.GetErrorTrace(err)
}
defer table.Close()
[...]
}
Expected behavior
It should find the file ๐
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Related code part should be the call of os.ReadDir(filepath.Dir(name))
in
Line 788 in e80f581
Tracking issue for:
By using file seekers, you can read specific parts of the file, which can allow you to read only one field of an entry at a time and match it with the search value. If the values match completely or partially, the corresponding line positions are returned.
When creating a column modification where the original column name and the target name (key) are equal, an index out of range exception occurs when trying to read the resulting dbf rows.
2022/08/29 15:32:51 Importing technicians...
2022/08/29 15:32:51 [importer] [0] interruption: runtime error: index out of range [-1]
exit status 0xc000013a
log.Print("Importing technicians...")
dbf, err := dbase.Open(filepath.Join(manager.config.DBPath, "DATABASE.dbf"), &dbase.Win1250Converter{})
if err != nil {
log.Panicf(err.Error())
}
defer dbf.Close()
dbf.SetTrimspacesDefault(true)
dbf.SetColumnModification(dbf.ColumnPos("TECHNR"), true, "ID", nil)
dbf.SetColumnModification(dbf.ColumnPos("Name"), true, "Name", nil) //crashes because of this line
for !dbf.EOF() {
row, err := dbf.Row()
if err != nil {
log.Panicf(err.Error())
}
dbf.Skip(1)
if row.Deleted {
continue
}
...
}
The original column names in the dbf table:
The target struct for reference:
type Technician struct {
ID uint64 `gorm:"primaryKey" json:"ID"`
Name string `gorm:"not null" json:"Name"`
CreatedAt time.Time
UpdatedAt time.Time
}
Reading the rows works perfectly fine when removing the line, but this action should not cause any problems...
Hi All
I'm getting an error when trying to write and then read a memo field.
Write call seems to succeed with no error thrown:
newNote := Note{
Text: "zHdVIxu2TuBmm/nY9CDbLotel+ToRIxApggFvFSCjsvU//uZXDjbXBfliSpkSubmzu/Imn7NTs6Y1zD7Q2rZHg==",
}
newRow, err := table.RowFromStruct(newD)
if err != nil {
panic(dbase.GetErrorTrace(err))
}
Add the new row to the database table.
err = newRow.Write()
if err != nil {
panic(dbase.GetErrorTrace(err))
}
But when I try to read the row I get:
panic: dbase-table-bytestorow-3:dbase-interpreter-parsememo-1:dbase-io-readmemo-5:parsing memo failed at column field: TEXT failed with error: INCOMPLETE
Here's my read code:
row, err := table.Row()
if err != nil {
panic(dbase.GetErrorTrace(err))
}
Full log
[dbase] [DEBUG] 2022/10/27 17:28:35 Reading row: 21 at offset: 1139
[dbase] [DEBUG] 2022/10/27 17:28:35 Converting row data (31 bytes) to row struct...
[dbase] [DEBUG] 2022/10/27 17:28:35 Reading memo block 367 at position 23488
[dbase] [DEBUG] 2022/10/27 17:28:35 Memo block header => text: true, length: 88
[dbase] [ERROR] 2022/10/27 17:28:35 dbase-io-readmemo-5:INCOMPLETE
[dbase] [ERROR] 2022/10/27 17:28:35 dbase-interpreter-parsememo-1:parsing memo failed at column field: TEXT failed with error: INCOMPLETE
[dbase] [ERROR] 2022/10/27 17:28:35 dbase-table-bytestorow-3:parsing memo failed at column field: TEXT failed with error: INCOMPLETE
[dbase] [DEBUG] 2022/10/27 17:28:35 Closing file: ./files/AAQBX.DBF
[dbase] [DEBUG] 2022/10/27 17:28:35 Closing related file: ./files/AAQBX.DBF
panic: dbase-table-bytestorow-3:dbase-interpreter-parsememo-1:dbase-io-readmemo-5:parsing memo failed at column field: TEXT failed with error: INCOMPLETE
I noticed that this only happens when "Text" is greater than 56 characters.
Thanks for all the work you guys do.
We are using this lib successfully to edit / insert into a DBF File Type: 0x30 (Visual FoxPro), windows-1252 encoding, that has a corresponding .CDX index file. We can insert into the DB, but we need to somehow update the index afterwards.
Is this already possible? If yes, please let me know how, otherwise, would it be possible to be implemented?
I would definitely help with contribution, but I'm not a go developer, ChatGPT is my friend for this small tool ๐
Thank you!
R
Please add in OpenTable
function reading from io.ReadCloser
interface.
I have files in archive and func return io.ReaderCloser. I think it will be useful.
Varchar can be up to 254 characters (always stored with constant size)
The _NullFlags column of the Table needs to be checked. (it's a hidden system column)
Tables that contain a varchar always contain the _NullFlags column.
Depending on, if the null-Flag for the varchar field is set, length is stored in the last byte of the field.
If the length is 0 the value should be interpreted as empty.
For each Varchar and Varbinary field, one bit, or "varlength" bit, is allocated in the last system field, which is a hidden field and stores the null status for all fields that can be null. If the Varchar or Varbinary field can be null, the null bit follows the "varlength" bit. If the "varlength" bit is set to 1, the length of the actual field value length is stored in the last byte of the field. Otherwise, if the bit is set to 0, length of the value is equal to the field size.
https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)#table-header-record-structure
the "Database export" link doesn't seem to work anymore
I have a struct with
...
PLANCREATE time.Time json:"PLANCREATE"
...
in the table some values in the row are 0 or null which fails to convert with the following error:
[dbase] [DEBUG] 2022/11/03 12:09:01 No encoding converter defined, falling back to default (interpreting)
[dbase] [DEBUG] 2022/11/03 12:09:01 Interpreting code page mark...
[dbase] [DEBUG] 2022/11/03 12:09:01 Code page: 0x03 => interpreted: 0x03
[dbase] [DEBUG] 2022/11/03 12:09:01 Reading row: 0 at offset: 7112
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row data (2021 bytes) to row struct...
[dbase] [DEBUG] 2022/11/03 12:09:01 Skipping 1 row/s, new position: 1
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row 0 to struct...
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row 0 to JSON...
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row 0 to map...
[dbase] [ERROR] 2022/11/03 12:09:01 dbase-table-tostruct-2:parsing time "0" as ""2006-01-02T15:04:05Z07:00"": cannot parse "0" as """
Is it possible to default the value in the struct when 0 or null is encountered in the table?
Thanks in advance!
I have a foxpro table with a column(EINGABEAM) that holds datetime information in the format MM/DD/YY HH:mm:ss AM/PM that i want to read.
Now when i am setting up the dbase reader
dbf, err := dbase.OpenTable(&dbase.Config{
Filename: filepath.Join(FILE_DIR, FILENAME),
TrimSpaces: true,
})
dbf.SetColumnModificationByName("EINGABEAM", &dbase.Modification{TrimSpaces: true, ExternalKey: "creationDate", Convert: imp.convDateToTimestamp})
and try to read its value:
val, _ := row.ValueByName("EINGABEAM")
log.Debugf("%+v", val)
val is always equal to
0001-01-01 00:00:00 +0000 UTC
no matter the actual value of the field (for example [01/13/2009 12:59:38 PM]).
The "EINGABEAM" field is of type DateTime with a width of 8.
Is this a bug or am i doing something wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.