distributedclocks / govector Goto Github PK

Sending an interface over vrpc causes a crash. For example the following types

type Operation interface {
	OperationType() OperationType
	GetMinerID() string
	GetFileName() string
}

type AppendOperation struct {
	Timestamp time.Time
	Filename  string
	Record    Record
	RecordNum uint16
	MinerId   string
}

Where we are sending and expecting Operation. This causes deserialization to fail with the following message:

panic: reflect.Set: value of type map[string]interface {} is not assignable to type blockchain.Operation

goroutine 5 [running]:
reflect.Value.assignTo(0x74a1c0, 0xc00040c2d0, 0x15, 0x7b4559, 0xb, 0x75cf60, 0xc0003ee1d0, 0xc00040c2d0, 0x0, 0x0)
	C:/Go/src/reflect/value.go:2239 +0x453
reflect.Value.Set(0x75cf60, 0xc0003ee1d0, 0x194, 0x74a1c0, 0xc00040c2d0, 0x15)
	C:/Go/src/reflect/value.go:1373 +0xae
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).interfaceValue(0xc00029e700, 0x75cf60, 0xc0003ee1d0, 0x194, 0x535792, 0xc00029e700)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:284 +0x163
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeInterfaceValue(0xc00029e700, 0x75cf60, 0xc0003ee1d0, 0x194, 0xc0003ee1d0, 0x194)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:256 +0x1ea
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.ptrDecoderFunc.func1(0xc00029e700, 0x71eb20, 0xc0003ee1d0, 0x16, 0x71eb20, 0x0)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:124 +0xca
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).DecodeValue(0xc00029e700, 0x71eb20, 0xc0003ee1d0, 0x16, 0xc0003ee1d0, 0x16)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:224 +0x93
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeInterfaceValue(0xc00029e700, 0x748420, 0xc00040c288, 0x194, 0x748420, 0x749800)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:267 +0xaa
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).DecodeValue(0xc00029e700, 0x748420, 0xc00040c288, 0x194, 0xc00040c288, 0x194)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:224 +0x93
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).decode(0xc00029e700, 0x7216e0, 0xc00040c288, 0xc0003ea110, 0xe)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:212 +0x182
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).Decode(0xc00029e700, 0xc00006f9d0, 0x1, 0x1, 0x0, 0x763718)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:98 +0x73
github.com/DistributedClocks/GoVector/govec.(*VClockPayload).DecodeMsgpack(0xc00040c270, 0xc00029e700, 0xc00040c270, 0x2b70338)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:184 +0xcf
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeCustomValue(0xc00029e700, 0x7636e0, 0xc00040c270, 0x16, 0xc00040c270, 0x16)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:174 +0x118
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.decodeCustomValueAddr(0xc00029e700, 0x772fe0, 0xc00040c270, 0x199, 0x772fe0, 0x452130)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode_value.go:132 +0x7c
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).DecodeValue(0xc00029e700, 0x772fe0, 0xc00040c270, 0x199, 0xc00040c270, 0x199)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:224 +0x93
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).decode(0xc00029e700, 0x7636e0, 0xc00040c270, 0x0, 0x40)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:212 +0x182
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.(*Decoder).Decode(0xc00029e700, 0xc00006fcc8, 0x1, 0x1, 0xc000036000, 0xc00006fcb8)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:98 +0x73
github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack.Unmarshal(0xc000286060, 0x52, 0x52, 0xc00006fcc8, 0x1, 0x1, 0x30, 0x772fe0)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/vendor/github.com/vmihailenco/msgpack/decode.go:36 +0xb1
github.com/DistributedClocks/GoVector/govec.defaultDecoder(0xc000286060, 0x52, 0x52, 0x7636e0, 0xc00040c270, 0x71eaa0, 0xc0000fe3e0)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:371 +0x85
github.com/DistributedClocks/GoVector/govec.(*GoLog).UnpackReceive(0xc000108080, 0x7b886f, 0x14, 0xc000286060, 0x52, 0x52, 0x71eb20, 0xc0003ee1d0, 0x1)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:553 +0x102
github.com/DistributedClocks/GoVector/govec/vrpc.(*RPCServerCodec).ReadRequestBody(0xc000202080, 0x71eb20, 0xc0003ee1d0, 0x71eb20, 0xc0003ee1d0)
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/vrpc/vrpc.go:138 +0xee
net/rpc.(*Server).readRequest(0xc0000943c0, 0x814ea0, 0xc000202080, 0xc00003c360, 0xc00003c3c0, 0xc00042df88, 0x6f897a, 0x5040408, 0xc0000524a0, 0x20, ...)
	C:/Go/src/net/rpc/server.go:569 +0x299
net/rpc.(*Server).ServeCodec(0xc0000943c0, 0x814ea0, 0xc000202080)
	C:/Go/src/net/rpc/server.go:465 +0x9e
created by github.com/DistributedClocks/GoVector/govec/vrpc.ServeRPCConn
	C:/Users/Denis/go/src/github.com/DistributedClocks/GoVector/govec/vrpc/vrpc.go:38 +0xb6

This likely because under the hood go might be serializing the interface as a map.

The library is missing tests. This makes it difficult to know when things break and whether or not it really works. We need tests for each of the API calls (unit tests), and integration tests for combinations of API calls.

We need to test the outputs written to Log files.
This would involve creating a mock library that can capture and test the outputs written to the Log File.

Something like this: https://medium.com/@matryer/5-simple-tips-and-tricks-for-writing-unit-tests-in-golang-619653f90742

Replace custom parsing with the flag package.

I can't import GoVector into the project and keep getting error where it is looking for

GoVector/govec/gopublisher.go:10:2: cannot find package "github.com/arcaneiceman/GoVector/broker" in any of: /usr/lib/go/src/pkg/github.com/arcaneiceman/GoVector/broker (from $GOROOT) /home/vm/gowork/src/github.com/arcaneiceman/GoVector/broker (from $GOPATH)

And then the same thing for gopublisher.go and other components. I added GoVector as git submodule to our project and did the import from the root of the project "project/GoVector/govec" where project is root directory of our project. As far as I understand GOROOT needs to be set when you install go into a custom directory, but I haven't. GOPATH is working fine (so far) and is pointing to $HOME/gowork where the project is as well. Should GoVector be installed under $GOPATH as well for this to work? I was trying to make it self contained by having it as submodule.

Some info:

• GoVector cloned using git submodule init
• project is at $GOPATH/src/project
• GoVector is at $GOPATH/src/project/GoVector
• the file that's importing GoVector is at $GOPATH/src/project/server
• import statement "project/GoVector/govec"

I would really appreciate some help. Might be something I am doing wrong. Thanks.

Currently a single broker is used for the test suite, this may cause state based test failures. If a broker could be initialized to listen on different ports for each test then the tests will be better encapsulated.

Possible issue with knowing which port to connect to as data is stored in a suite variable. Tests do not appear to be run sequentially. This seems to be a function of the go testing package rather than go-check.

Issue:
When I have several clients logging events to the same log file, only the first client of the library has an entry with a vector clock starting at 1. The rest of the clients do not have an entry where the vector clock starts at 1 for them. As a result, when I upload a log file to https://bestchai.bitbucket.io/shiviz/ I get an error saying

Logical clock values for each host must start at 1.
The clock value for the first event for host "34784" is 2.

Code used to obtain the problem:
This is the structure of my project

• network
• • network.go
• client.go
• server.go

These extensions of these files should be changed to '.go'
client.txt
network.txt
server.txt

# How to run this project
$ server.go # in one terminal, leave it running
$ client.go <IP:port of the server> # run this several times, in different terminals 

Proposed fix: If the log file exists, we still need to log some kind of event, which will result in having an entry with a vector clock starting at 1.
patchFix.txt
Update: Actually I don't know enough about the project to be sure that this will work. In particular, Im not sure why we have different execution numbers, and do entries for new processes must start with vector clock 1?

Hello,

Say I am very pleased with the logging and believe everything works perfectly. I don't want to change my code but I now longer want to log large number of files for my system.

Is there a method/flag in the library to disable logging completely? I see there are some to print out the logging to the screen, but none to disable.

At the moment the broker code has a single web-socket. We need two, one for servicing publishers, and a second one for servicing subscribers.

Formulate the subscriber API version .1 This should include authentication/nonce bits for setting up the connection. This API should reflect client concerns. Some things that it may contain are: GetOldEvents (before subscriber joiner), CreateFilter (only notify subscriber of events of particular nature), RemoveFilter (disable the filtering), PauseStream (pause the publishing of events to this subscriber), UnPauseStream (resume publishing of events to this subscriber -- should events that were on pause be sent?).

A complete API would need to be specified as a JSON RPC set of method calls, and an example client (in Go, or JS) that would interface with the server-side API.

Currently successfully instrumenting a directory using GoVector yields output looking like this:

$ GoVector -dir="example"
Source Built
Source Built
Wrappers Built
var conn net.PacketConn
var conn net.PacketConn

It'd be great to make the result more clear to help new users understand what's happing. It could look something like this:

$ GoVector -dir="example"
Reading example/main.go
Reading example/utils.go
examples/main.go:34 net.ReadFrom instrumented
examples/main.go:55 net.WriteTo instrumented
Overwriting original files
SUCCESS 2/2 detected network calls instrumented

Need to wrap the ServeConn into a go routine so that you can (1) service multiple subscribers, and (2) do this asynchronously.

Example of doing this with RPC over a TCP connection:
https://www.artima.com/weblogs/viewpost.jsp?thread=333589

Refactor GoPublisher to its own .go file and update govec.go accordingly.

The broker will need to be parameterized in a number of ways:

• IP/port for subscribers (required)
• IP/port for publishers (required)
• file for file-subscriber (optional)
• token for authenticating subscribers (optional)
• token for authenticating publishers (optional)
• IP origin authentication option (optional)

Instead of architecting a command line madness, it would be great to push these into a config file and use an existing config file library for reading these options.

The broker uses the console to print verbose logging messages. Create a separate file to record these messages (separate from the vector clock log file intended for ShiViz).

GoLog's initialize functions (Initialize and InitializeMutipleExecutions) do not accept parameters for GoVec settings.

Create a struct that holds the appropriate settings (gv.printonscreen (ShouldYouSeeLoggingOnScreen) gv.VConWire (ShouldISendVectorClockonWire),gv.debugmode (Debug), gv.realtime = false etc), modify the initialize functions to accept this struct and modify the GoLog struct to store and use the configuration struct.

Hi!

I've used this library in college for a little proyect and all the teammates I meet had the same problem I had.

In the govec program, inside the govec folder, line 208 has a bug where you passed 3 arguments to the function dec.Decode(). An error pop up saying how it's supposed to be and, actually, in the line 183 is correctly used. So i changed it to err = dec.Decode(&d.Payload) and it works.

I see is an easy fix, but using an IDE (for example Goland) it warns you and tries to prevent you from changing any file that is an external library. Maybe, for an even more amateur programmer than me, this could be the end of the work for a while.

Index.html was modeled on a chat server. Should be simplified and updated to reflect current usage.

Both publishers and subscribers are being served by the same RPC server (the default), this may mean that a subscriber can make publisher RPC calls and vice versa.

Create separate servers so that only publisher RPC calls are available on the publisher port and only subscriber rpc calls are available on the subscriber port.

Refactor logging with a new method that checks the appropriate flags and then logs the requested level of information.

There are a number of if statements checking flags that should be refactored in this way.

Regex filters on PID, VClock, message receipt time.

Local message compression filter to compress a sequence of local messages from a single host/PID

Write tests for an end to end application of the system. Currently, all we have is the code found in Client Server code found in /example.

Currently messages are printed to console.

Allow for a log file that is used for GoVec messages only (such as error and debug messages).

Formulate the publisher API version .1 This should include authentication/nonce bits for setting up the connection. It should also include a set of event types that are relevant to logging in a distributed context: LogEvent, JoinSystem (implicit?), LeaveSystem, others?

A complete API would need to be specified as a JSON RPC set of method calls, and an example client that would interface with the server-side API.

Refactor logging with a new method that checks the appropriate flags and then logs the requested level of information.

Must also update the Broker initialization in broker.go to accept debugging flags to turn on/off log messages/file logging.

Unfortunately there appears to be a bug in the vector clock comparison implementation:

This simple test will fail:

func TestCompareDifferentLengths(t *testing.T) {
	n1 := New()
	n2 := New()

	n1.Set("a", 1)
	n1.Set("b", 1)

	n2.Set("b", 1)
	n2.Set("c", 1)
	n2.Set("d", 1)

	if n1.Compare(n2, Descendant) {
		failComparison(t, "Clocks are defined as Descendant: n1 = %s | n2 = %s", n1, n2)
	}
}

The comparison function will detect that (B:1, C:1, D:1) is a descendant of (A:1, B1:), although they are concurrent.
I don't have a fix for this issue, but I'd like to warn future users of the package.
There does not seem to be any ongoing maintenance.
For now, we have created a fork of just the vclock implementation: https://github.com/OpenFogStack/vclock

Formulate the subscriber API version .1 This should include authentication/nonce bits for setting up the connection. It should also include a set of event types that are relevant to advertising set of system events to the client. Roughly this include the same set of events as in the publisher API: NewEvent, ProcessJoined, ProcessLeft,

A complete API would need to be specified as a JSON RPC set of method calls, and an example client (in Go, or JS) that would interface with the server-side API.

Update LocalMessage and NetworkMessage to hold a byte representation of the vclock.

Update relevant portions of govec.go and gopublishers.go to handle this change.

if registerPublisher tries to register a nonce that already exists it should return an error to be dealt with in setupPubManagerTCP, otherwise the pubmanager will be shutdown for all potential new publishers.

When a publisher disconnects it has maybe dropped out of the distributed system. It would be good to log that as an event with an appropriate vector clock entry.

The vector clock could either be reconstructed from the most recent message received or messages could be refactored to record which publisher they came from if we wanted to record the last seen vclock.

Code I am using:

package main

import (
	"github.com/DistributedClocks/GoVector/govec"
)

func main() {
	LoggerClient := govec.InitGoVectorMultipleExecutions("2", "test")
	LoggerClient.PrepareSend("Sending", "blabla")
}

Problem:
I get the following error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12dd1ad]

goroutine 1 [running]:
github.com/DistributedClocks/GoVector/govec.(*GoLog).PrepareSend(0xc420134100, 0x1393dde, 0x7, 0x1316880, 0x13d6600, 0x0, 0xc420125f70, 0x12de28e)
        /Users/annygakh/go/src/github.com/DistributedClocks/GoVector/govec/govec.go:507 +0x25d
main.main()
        /Users/annygakh/go/src/test-shiviz/test.go:12 +0x93
exit status 2

Possible fix:
It seems like there is missing initialization of private fields gv.encodingStrategy (and gv.decodingStrategy). The fields are initialized in InitGoVector, but not in InitGoVectorMultipleExecutions (which is what I am using). Attached is a proposed fix.

patch.diff.txt

Currently the FileSub opens and closes the log file every time it writes a message. Change this to leave the file open and close it at the appropriate time.

This could be part of a deferred method call in the Broker to close all the connections to publishers and subscribers when the broker shuts down.