Comments (11)
Ok. Generating a schema in Janus, we see that it uses Date
and not Timestamp
. Here is an example that creates a simple schema and implements a custom Date GraphSON serializer. I believe that this is the most correct way to handle Janus dates with Goblin:
import asyncio
import datetime
import goblin
from aiogremlin import Cluster
from aiogremlin.gremlin_python.driver import serializer
from aiogremlin.gremlin_python.process.traversal import P
from aiogremlin.gremlin_python.structure.io import graphson
def get_hashable_id(val):
if isinstance(val, dict) and "@type" in val and "@value" in val:
if val["@type"] == "janusgraph:RelationIdentifier":
val = val["@value"]["value"]
return val
class DateSerializer:
def dictify(self, obj, writer):
# Java timestamp expects miliseconds
ts = round(obj.timestamp() * 1000)
return graphson.GraphSONUtil.typedValue('Date', ts)
class DateDeserializer:
def objectify(self, ts, reader):
# Python timestamp expects seconds
dt = datetime.datetime.fromtimestamp(ts / 1000.0)
return dt
reader = graphson.GraphSONReader({'g:Date': DateDeserializer()})
writer = graphson.GraphSONWriter({datetime.datetime: DateSerializer()})
message_serializer = serializer.GraphSONMessageSerializer(reader=reader,
writer=writer)
loop = asyncio.get_event_loop()
cluster = loop.run_until_complete(
Cluster.open(loop, message_serializer=message_serializer))
app = goblin.Goblin(cluster)
class DateTime(goblin.abc.DataType):
def validate(self, val):
if not isinstance(val, datetime.datetime):
raise goblin.exception.ValidationError(
"Not a valid datetime.datetime: {}".format(val))
return val
def to_ogm(self, val):
return super().to_ogm(val)
def to_db(self, val):
return super().to_db(val)
class Event(goblin.Vertex):
name = goblin.Property(goblin.String)
datetime = goblin.Property(DateTime)
app.register(Event)
async def create_schema():
client = await cluster.connect()
schema_msg = """mgmt = graph.openManagement()
datetime = mgmt.makePropertyKey('datetime').dataType(Date.class).cardinality(Cardinality.SINGLE).make()
mgmt.commit()"""
await client.submit(schema_msg)
async def go():
session = await app.session()
# Create an event with a datetime property
event1 = Event()
event1.name = 'event1'
event1.datetime = datetime.datetime.now()
# Get a timestamp for comparisons
await asyncio.sleep(0.001)
ts = datetime.datetime.now()
await asyncio.sleep(0.001)
# Create an event with a later datetime attribute
event2 = Event()
event2.name = 'event2'
event2.datetime = datetime.datetime.now()
# Add event verts to DB
session.add(event1, event2)
await session.flush()
# Query based on datetime
earlier_event = await session.g.V().has('datetime', P.lt(ts)).next()
print("{} occured at {}".format(earlier_event.name, earlier_event.datetime))
later_event = await session.g.V().has('datetime', P.gt(ts)).next()
print("{} occured at {}".format(later_event.name, later_event.datetime))
loop.run_until_complete(create_schema())
loop.run_until_complete(go())
loop.run_until_complete(app.close())
from goblin.
I'll have to look into this a bit. Hopefully I can do it this weekend. I am considerably busier than usual right now, as I am in the process of submitted a dissertation and moving 2000 miles.
from goblin.
I look forward to any insights you can offer. Obviously, timestamp data can be saved in the graph as integers (or floats), but it would be ideal to have time data saved as timestamp type.
from goblin.
Sure, if you want to do that, it is easy to create a new data type. Simply inherit from abc.DataType
and implement the validate
, to_db
, and to_ogm
methods. Here is a simple example of an IP address data type:
class IPAddress(abc.DataType):
def to_ogm(self, val):
return ipaddress.ip_address(val)
def to_db(self, val=None):
return str(super().to_db(val=val))
def validate(self, address):
if not address:
return
try:
val = ipaddress.ip_address(address)
except ValueError as e:
raise exception.ValidationError('Not a valid IP address') from e
return val
from goblin.
Quick question, do you want to traverse element based on when they were inserted into the graph? Or do you want to add a datetime to an element and use it as the basis for comparison in traversal? Or both..?
from goblin.
Both would be nice, but the latter is more important to me. One option might be to use numpy datetime objects to hold time information in a property for a vertex or edge.
from goblin.
Here is a quick example of how to do this using a custom DateTime datatype. Here, the user can add a datetime to a Goblin, which is stored in the graph as Posix timestamp. You can then use any type of comparison you want to query the graph. This approach could be easily extended to traverse timestamps generated by the db, or to use timezone aware datetimes, or datetimes from other modules like numpy. Hopefully this helps you get going. Please let me know if you have any questions:
import asyncio
import datetime
import goblin
from aiogremlin.gremlin_python.process.traversal import P
def get_hashable_id(val):
if isinstance(val, dict) and "@type" in val and "@value" in val:
if val["@type"] == "janusgraph:RelationIdentifier":
val = val["@value"]["value"]
return val
loop = asyncio.get_event_loop()
app = loop.run_until_complete(
goblin.Goblin.open(loop, get_hashable_id=get_hashable_id))
class DateTime(goblin.abc.DataType):
def validate(self, val):
if not isinstance(val, datetime.datetime):
raise goblin.exception.ValidationError(
"Not a valid datetime.datetime: {}".format(val))
return val
def to_ogm(self, val):
return datetime.datetime.fromtimestamp(val)
def to_db(self, val):
return val.timestamp()
class Event(goblin.Vertex):
name = goblin.Property(goblin.String)
datetime = goblin.Property(DateTime)
app.register(Event)
async def go():
session = await app.session()
# Create an event with a datetime property
event1 = Event()
event1.name = 'event1'
event1.datetime = datetime.datetime.now()
# Get a timestamp for comparisons
ts = datetime.datetime.now().timestamp()
# Create an event with a later datetime attribute
event2 = Event()
event2.name = 'event2'
event2.datetime = datetime.datetime.now()
# Add event verts to DB
session.add(event1, event2)
await session.flush()
# Query based on datetime
earlier_event = await session.g.V().has('datetime', P.lt(ts)).next()
print("{} occured at {}".format(earlier_event.name, earlier_event.datetime))
later_event = await session.g.V().has('datetime', P.gt(ts)).next()
print("{} occured at {}".format(later_event.name, later_event.datetime))
loop.run_until_complete(go())
loop.run_until_complete(app.close())
from goblin.
I had a minute to reread this thread, and I realized I only gave you a partial answer. The above code is an easy workaround to get going. To send the timestamps to the server as timestamp type, you would need to serialize a datetime object to a graphson2 timestamp type. As gremlin_python doesn't currently implement a serializer for timestamp, you would have to implement a simple custom serializer class, and pass it to the Cluster
used by the Goblin
app. I think the end result of this solution is pretty similar to the one I posted above at least for the time being, but it is an issue that needs to be addressed in gremlin-python at some point, so it is worth experimenting with it here. I'll try to post an example later tonight or tomorrow morning.
from goblin.
I implemented an example using custom GraphSON de/serializers that works great with TinkerGraph. However, when I run it against Janus I get the following error:
GremlinServerError: 500: Property value [2017-08-24 19:18:32.865] is of type class java.sql.Timestamp is not supported
I would have to look into Janus graph support for timestamp to figure exactly what is going on, maybe I can ask around during work hours tomorrow. For reference, the above code modified to use the custom serializers is as follows. Since it works with the reference implementation, it should, in theory, work with Janus as well:
import asyncio
import datetime
import goblin
from aiogremlin import Cluster
from aiogremlin.gremlin_python.driver import serializer
from aiogremlin.gremlin_python.process.traversal import P
from aiogremlin.gremlin_python.structure.io import graphson
def get_hashable_id(val):
if isinstance(val, dict) and "@type" in val and "@value" in val:
if val["@type"] == "janusgraph:RelationIdentifier":
val = val["@value"]["value"]
return val
class DateTimeSerializer:
def dictify(self, obj, writer):
# Java timestamp expects miliseconds
ts = round(obj.timestamp() * 1000)
return graphson.GraphSONUtil.typedValue('Timestamp', ts)
class DateTimeDeserializer:
def objectify(self, ts, reader):
# Python timestamp expects seconds
dt = datetime.datetime.fromtimestamp(ts / 1000.0)
return dt
reader = graphson.GraphSONReader({'g:Timestamp': DateTimeDeserializer()})
writer = graphson.GraphSONWriter({datetime.datetime: DateTimeSerializer()})
message_serializer = serializer.GraphSONMessageSerializer(reader=reader,
writer=writer)
loop = asyncio.get_event_loop()
cluster = loop.run_until_complete(
Cluster.open(loop, message_serializer=message_serializer))
app = goblin.Goblin(cluster)
class DateTime(goblin.abc.DataType):
def validate(self, val):
if not isinstance(val, datetime.datetime):
raise goblin.exception.ValidationError(
"Not a valid datetime.datetime: {}".format(val))
return val
# Note that these methods are different than in the previous example.
def to_ogm(self, val):
return super().to_ogm(val)
def to_db(self, val):
return super().to_db(val)
class Event(goblin.Vertex):
name = goblin.Property(goblin.String)
datetime = goblin.Property(DateTime)
app.register(Event)
async def go():
session = await app.session()
# Create an event with a datetime property
event1 = Event()
event1.name = 'event1'
event1.datetime = datetime.datetime.now()
# Get a timestamp for comparisons
await asyncio.sleep(0.001)
ts = datetime.datetime.now() # here we don't need to convert to timestamp
await asyncio.sleep(0.001)
# Create an event with a later datetime attribute
event2 = Event()
event2.name = 'event2'
event2.datetime = datetime.datetime.now()
# Add event verts to DB
session.add(event1, event2)
await session.flush()
# Query based on datetime
earlier_event = await session.g.V().has('datetime', P.lt(ts)).next()
print("{} occured at {}".format(earlier_event.name, earlier_event.datetime))
later_event = await session.g.V().has('datetime', P.gt(ts)).next()
print("{} occured at {}".format(later_event.name, later_event.datetime))
loop.run_until_complete(go())
loop.run_until_complete(app.close())
It is also of note that in my previous example, the custom data type should probably convert the between the Python (seconds from epoch) and Java (miliseconds from epoch) timestamps.
from goblin.
It occurred to me this morning that Janus is probably just like Titan, so the real way to control types in the db is through the schema definition. Glancing at the docs it appears that this is correct. I'll put up an example of this a bit later.
from goblin.
@John-Boik @davebshow
Hello guys!
Do you have a demo about the Goblin for Front end development Showcase?
from goblin.
Related Issues (20)
- Reference instance to test goblin against HOT 3
- How to change version of Goblin OGM Serializer for Janusgraph (TinkerPop 3.2.6) HOT 3
- TypeError: unhashable type: 'dict' when creating an Edge HOT 10
- goblin does not work with aiohttp>=3.0.0 HOT 5
- Question about using a multi-value/set VertexProperty HOT 6
- Vertex property gets changed from Gremlin Console but not from Python/Goblin script HOT 2
- goblin not working with aiohttp (server) HOT 2
- Mocking JanusGraph Connection for unit tests
- Add method to easily remove Vertex HOT 1
- The project appears idle, can I take over? HOT 10
- Problem with Edge data type HOT 5
- App/Cluster configuration using URL HOT 3
- Schema generator HOT 3
- Belated License change HOT 5
- Question: Extent to which OO modeling is supported in Goblin HOT 7
- How to use mixed index in goblin? HOT 4
- AttributeError: AsyncGraphTraversal.last_traverser HOT 1
- [Proposal] - OGM distinct from driver HOT 7
- Ability to set vertex or edge ID HOT 5
- Unexpected results when updating property HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goblin.