A code page is a set of characters specific to a language or hardware platform. Accented characters are not represented by the same values across platforms and code pages. In addition, some characters available in one code page are not available in another.

• U.S. MS-DOS x01
• International MS-DOS x02
• Eastern European MS-DOS x64
• Nordic MS-DOS x66
• Russian MS-DOS x65
• Thai Windows x7C
• Eastern European Windows xC8
• Russian Windows xC9
• Windows ANSI x03
• Greek Windows xCB
• Turkish Windows xCA
• Hebrew Windows x7D
• Arabic Windows x7E
• Mazovia (Polish) MS-DOS x69
• Greek MS-DOS (437G) x6A
• Kamenicky (Czech) MS-DOS x68
• Japanese Windows x7B
• Chinese (PRC, Singapore) Windows x7A
• Korean Windows x79
• Chinese (Hong Kong SAR, Taiwan) Windows x78
• Turkish MS-DOS x6B
• Icelandic MS-DOS x67
• Standard Macintosh x04
• Greek Macintosh x98
• Russian Macintosh x96

https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/8t45x02s(v=vs.71)

When new entries are written, the NullFlag column is not written. So that there are no variables and zero fields in written entries.

Visual FoxPro uses tables to store data that defines different file types. The following list includes the file types that are saved as tables:

• Database (.dbc)
• Form (.scx)
• Label (.lbx)
• Menu (.mnx)
• Project (.pjx)
• Report (.frx)
• Table (.dbf)
• Visual class library (.vcx)

You can use and browse these files in the same way that you browse any table file because these files are actually tables.

https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)

Currently, tables can be read and written, but for this the files must always already exist. It would be nice if you could also create new files with the package.

I'm not sure if there is an issue with my Go environment on macOS, but when I try to use the module (for example by running the examples), I get syscall.Handle errors:

/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:13:27: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:14:27: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:121:28: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:149:31: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:151:36: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:199:32: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/table.go:76:40: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/table.go:88:32: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/interpreter.go:98:33: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/interpreter.go:104:36: undefined: syscall.Handle
/Users/macuser/go/pkg/mod/github.com/!valentin-!kaiser/go-dbase/[email protected]/io.go:151:36: too many errors

Is there a known issue with running this module on macOS? (M1, Ventura 13.1)

The "TrimSpaces" option in the dbase.Config is only used in the ToStruct -> ToMap functions
When getting the values directly the spaces remain in the result

Describe the bug
On windows my file isn't recognized and throwing dbase-io-windows-findfile-1. My code uses filepath.Join(pi.Config.DBPath, 'daten', 'my-table.DBF') which works on my linux clients. However on windows the path with the config prefix does not work. I printed the path Y:\some-folder\daten\my-table.DBF already and checked if it exists by entering it to win+r and everything worked as expected. I am a bit lost where the issue could come from.

To Reproduce
Example code to reproduce the behavior:

package main

import (
    "fmt"
    "github.com/Valentin-Kaiser/go-dbase"
)

func main() {
    configPath := 'Y:\somefolder'
    p := filepath.Join(configPath, 'daten', 'mytable.DBF')


      table, err := dbase.OpenTable(&dbase.Config{
		Filename:   p,
		TrimSpaces: true,
	})
	if err != nil {
		return dbase.GetErrorTrace(err)
	}
	defer table.Close()

       [...]
}

Expected behavior
It should find the file 😅

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: windows 10 amd64
• Go version: 1.21.2
• Package version: v1.11.9

Additional context

Related code part should be the call of os.ReadDir(filepath.Dir(name)) in

Tracking issue for:

• https://github.com/Valentin-Kaiser/go-dbase/security/code-scanning/1

By using file seekers, you can read specific parts of the file, which can allow you to read only one field of an entry at a time and match it with the search value. If the values match completely or partially, the corresponding line positions are returned.

When creating a column modification where the original column name and the target name (key) are equal, an index out of range exception occurs when trying to read the resulting dbf rows.

2022/08/29 15:32:51 Importing technicians...
2022/08/29 15:32:51 [importer] [0] interruption: runtime error: index out of range [-1]
exit status 0xc000013a

log.Print("Importing technicians...")

dbf, err := dbase.Open(filepath.Join(manager.config.DBPath, "DATABASE.dbf"), &dbase.Win1250Converter{})
	if err != nil {
		log.Panicf(err.Error())
	}
	defer dbf.Close()

	dbf.SetTrimspacesDefault(true)
	dbf.SetColumnModification(dbf.ColumnPos("TECHNR"), true, "ID", nil)
	dbf.SetColumnModification(dbf.ColumnPos("Name"), true, "Name", nil) //crashes because of this line

	for !dbf.EOF() {
		row, err := dbf.Row()
		if err != nil {
			log.Panicf(err.Error())
		}
		dbf.Skip(1)

		if row.Deleted {
			continue
		}
...
}

The original column names in the dbf table:

The target struct for reference:

type Technician struct {
	ID        uint64 `gorm:"primaryKey" json:"ID"`
	Name      string `gorm:"not null" json:"Name"`
	CreatedAt time.Time
	UpdatedAt time.Time
}

Reading the rows works perfectly fine when removing the line, but this action should not cause any problems...

Hi All

I'm getting an error when trying to write and then read a memo field.

Write call seems to succeed with no error thrown:

newNote := Note{
	Text: "zHdVIxu2TuBmm/nY9CDbLotel+ToRIxApggFvFSCjsvU//uZXDjbXBfliSpkSubmzu/Imn7NTs6Y1zD7Q2rZHg==",
}

newRow, err := table.RowFromStruct(newD)
if err != nil {
	panic(dbase.GetErrorTrace(err))
}

Add the new row to the database table.
err = newRow.Write()
if err != nil {
	panic(dbase.GetErrorTrace(err))
}

But when I try to read the row I get:

panic: dbase-table-bytestorow-3:dbase-interpreter-parsememo-1:dbase-io-readmemo-5:parsing memo failed at column field: TEXT failed with error: INCOMPLETE

Here's my read code:

row, err := table.Row()
if err != nil {
	panic(dbase.GetErrorTrace(err))
}

Full log

[dbase] [DEBUG] 2022/10/27 17:28:35 Reading row: 21 at offset: 1139
[dbase] [DEBUG] 2022/10/27 17:28:35 Converting row data (31 bytes) to row struct...
[dbase] [DEBUG] 2022/10/27 17:28:35 Reading memo block 367 at position 23488
[dbase] [DEBUG] 2022/10/27 17:28:35 Memo block header => text: true, length: 88
[dbase] [ERROR] 2022/10/27 17:28:35 dbase-io-readmemo-5:INCOMPLETE
[dbase] [ERROR] 2022/10/27 17:28:35 dbase-interpreter-parsememo-1:parsing memo failed at column field: TEXT failed with error: INCOMPLETE
[dbase] [ERROR] 2022/10/27 17:28:35 dbase-table-bytestorow-3:parsing memo failed at column field: TEXT failed with error: INCOMPLETE
[dbase] [DEBUG] 2022/10/27 17:28:35 Closing file: ./files/AAQBX.DBF
[dbase] [DEBUG] 2022/10/27 17:28:35 Closing related file: ./files/AAQBX.DBF
panic: dbase-table-bytestorow-3:dbase-interpreter-parsememo-1:dbase-io-readmemo-5:parsing memo failed at column field: TEXT failed with error: INCOMPLETE

I noticed that this only happens when "Text" is greater than 56 characters.

Thanks for all the work you guys do.

We are using this lib successfully to edit / insert into a DBF File Type: 0x30 (Visual FoxPro), windows-1252 encoding, that has a corresponding .CDX index file. We can insert into the DB, but we need to somehow update the index afterwards.

Is this already possible? If yes, please let me know how, otherwise, would it be possible to be implemented?

I would definitely help with contribution, but I'm not a go developer, ChatGPT is my friend for this small tool 🙂

Thank you!
R

Please add in OpenTable function reading from io.ReadCloser interface.
I have files in archive and func return io.ReaderCloser. I think it will be useful.

Varchars need special handling

Varchar can be up to 254 characters (always stored with constant size)

To retrieve the size of the Varchar

The _NullFlags column of the Table needs to be checked. (it's a hidden system column)
Tables that contain a varchar always contain the _NullFlags column.
Depending on, if the null-Flag for the varchar field is set, length is stored in the last byte of the field.
If the length is 0 the value should be interpreted as empty.

For each Varchar and Varbinary field, one bit, or "varlength" bit, is allocated in the last system field, which is a hidden field and stores the null status for all fields that can be null. If the Varchar or Varbinary field can be null, the null bit follows the "varlength" bit. If the "varlength" bit is set to 1, the length of the actual field value length is stored in the last byte of the field. Otherwise, if the bit is set to 0, length of the value is equal to the field size.
https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)#table-header-record-structure

the "Database export" link doesn't seem to work anymore

I have a struct with
...
PLANCREATE time.Time json:"PLANCREATE"
...

in the table some values in the row are 0 or null which fails to convert with the following error:

[dbase] [DEBUG] 2022/11/03 12:09:01 No encoding converter defined, falling back to default (interpreting)
[dbase] [DEBUG] 2022/11/03 12:09:01 Interpreting code page mark...
[dbase] [DEBUG] 2022/11/03 12:09:01 Code page: 0x03 => interpreted: 0x03
[dbase] [DEBUG] 2022/11/03 12:09:01 Reading row: 0 at offset: 7112
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row data (2021 bytes) to row struct...
[dbase] [DEBUG] 2022/11/03 12:09:01 Skipping 1 row/s, new position: 1
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row 0 to struct...
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row 0 to JSON...
[dbase] [DEBUG] 2022/11/03 12:09:01 Converting row 0 to map...
[dbase] [ERROR] 2022/11/03 12:09:01 dbase-table-tostruct-2:parsing time "0" as ""2006-01-02T15:04:05Z07:00"": cannot parse "0" as """

Is it possible to default the value in the struct when 0 or null is encountered in the table?

Thanks in advance!

Visual Foxpro / dbase data types:

• W - Blob
• C – Character
• C – Character (binary)
• Y – Currency
• B – Double
• D – Date
• T – DateTime
• F – Float
• G – General
• I – Integer
• L – Logical
• M – Memo
• M – Memo (binary)
• N – Numeric
• P – Picture
• Q - Varbinary
• V - Varchar
• V - Varchar (binary)

https://learn.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)#table-header-record-structure

I have a foxpro table with a column(EINGABEAM) that holds datetime information in the format MM/DD/YY HH:mm:ss AM/PM that i want to read.

Now when i am setting up the dbase reader

dbf, err := dbase.OpenTable(&dbase.Config{
		Filename:   filepath.Join(FILE_DIR, FILENAME),
		TrimSpaces: true,
	})
dbf.SetColumnModificationByName("EINGABEAM", &dbase.Modification{TrimSpaces: true, ExternalKey: "creationDate", Convert: imp.convDateToTimestamp})

and try to read its value:

val, _ := row.ValueByName("EINGABEAM")
		log.Debugf("%+v", val)

val is always equal to
0001-01-01 00:00:00 +0000 UTC
no matter the actual value of the field (for example [01/13/2009 12:59:38 PM]).

The "EINGABEAM" field is of type DateTime with a width of 8.
Is this a bug or am i doing something wrong?