The server is https://github.com/manzt/hg is largely copied from here. It would be good to extract this into its own reusable package.

Is there a way to provide a constant value to y-axis in the bar chart (and others as well)? For example, in the gallery example of bar chart, is it possible to a set constant value 0.0005 to y instead of reading specifying it from the data object?

I am wondering if it makes sense to expose gos.Track() only to users instead of the additional gos.PartialTrack(). I think whenever a user uses overlay alignment with tracks, we can internally consider this as PatialTrack so that users do not have to manually select between SingleTrack and PartialTrack.

bar_overview = bar_base.properties(
    width=300, height=300,
    alignment='overlay',
    tracks=[
        gos.Track(), // since overlay alignment is used with tracks, this is PartialTrack
        gos.Track().mark_brush().encode(x=gos.Channel(linkingId='detail-1')),
        gos.Track().mark_brush().encode(x=gos.Channel(linkingId='detail-2'))
    ]
)

Hi, this is my code:
`import streamlit as st
import gosling as gos
import streamlit_gosling as st_gos

if name == "main":
st.set_page_config(layout="wide")
col1, col2 = st.columns([1, 5])

@st.cache_resource
def chart():
    data1= gos.bigwig(url="/Users/violacimatti/Desktop/MB12_H3K27me3.bw", column="position", value="peak")

    track1= gos.Track(data1).mark_bar().encode(
            x=gos.X('position:G'),
            y=gos.Y('peak:Q', axis='right'),
            tooltip=[gos.Tooltip('position:G'), gos.Tooltip('peak:Q')],
        ).properties(layout="linear", width=1000, height=200, id='track-1', experimental={"mouseEvents": True}, title='')

    data2=gos.csv(
            url='/Users/violacimatti/Desktop/mod.bed', 
            headerNames=['chrom','Start','End','name','score','strand'],
            chromosomeField='chrom',
            genomicFields=['Start', 'End'],
            separator='\t'
            )

    base= gos.Track(data2).transform_displace(
        method='pile',
        newField='pileup',
        boundingBox=dict(
            startField='start',
            endField='end',
            padding=5
        ),
    ).transform_json_parse(
        field='subsitutions',
        genomicField='pos',
        baseGenomicField='start',
        genomicLengthField='length'
    ).properties(height=500)
    
    track2= base.mark_rect().encode(
                x=gos.X('start:G',scale=gos.scale(zero=False)),
                xe=gos.Xe('end:G'),
                row=gos.Row('pileup:N', padding=0.2),
                color=gos.value('red'),
            ).properties(width=1000, height=600, id='track-2', experimental={"mouseEvents": True}, title='')
  
  
    final= gos.stack(track1, track2).properties(xDomain=gos.GenomicDomain(chromosome="chr3", interval=[103385937, 103385937+10]), layout="linear", width=1000, height=800, id='track-3', experimental={"mouseEvents": True}, title='')
    return final 


with col1:
    st.header('Streamlit-Gosling')
    chr = st.selectbox('zoom to chromosome', [str(i) for i in range(1, 20)])
    start = st.number_input('Start Position', min_value=1, value=1, step=1)
    end = st.number_input('End Position', min_value=1, value=1, step=1)

eventType = 'zoomTo'
api = {"action": 'zoomTo', 'viewId': 'track-1', 'position': f'chr{chr}:{start}-{end}'}

with col2:
    result = st_gos.from_gos(spec=chart(), id='id', height=800, eventType=eventType, api=api)

with col1:
    st.write(f'You {eventType}: {result}')

`

I don't know why but the second track is empty when I visualize it. Also if you need it this is the bed file I would like to visualize:
chrom Start End name score strand
11 63063076 63063077 TRA00060190 . +
15 95051084 95051085 TRA00101969 . -
3 103385937 103385938 DEL00002431 . -
4 83009692 83009693 DEL00003020 . -
6 16644483 16644484 INV00005303 . -
X 148068900 148068901 TRA00166442 . +

Hi,
I am having trouble getting points to pile up. I tried something like this:

transform = gos.DisplaceTransform(boundingBox={"startField": "start:G", "endField": "end:G"},
                                  method=gos.DisplacementType("pile"),
                                  type="displace", newField="start", maxRows=20)
pile = gos.Displacement(gos.DisplacementType('pile'), padding=3.5)

seq_track = gos.Track(data, displacement=pile, dataTransform=[transform]).properties(height=150)
snp_mark = seq_track.mark_point(
).encode(
    x=gos.X("start:G", axis="bottom", linkingId='detail-1'),
    color=gos.Color("qval:Q", legend=True),
).properties(
    layout="linear", width=900, height=80,
    assembly=current_assembly)

The above code passes validation but doesn't render anything if I pass displacement or dataTransform to Track. I have tried using with dataTransform parameter only, but no luck. One thing that I am unsure of is what the value of newField parameter in gosling.DataTransform class is supposed to.

In case you need the full code, here it is:

Currently using the data-server is pretty magical. We could make this a bit more explicit for users:

import gosling as gos

data = gos.csv("./data/data.csv", ...) # throws if local file is detected

import gosling as gos
gos.enable_server() # run once

data = gos.csv("./data/data.csv", ...) # works

This would allow us to manage the server resources as well:

import gosling as gos

with gos.enable_server():
    data = gos.csv("./data/data.csv", ...)
    ...

data = gos.csv("./data/data.csv", ...) # throws

Explore approach used in https://github.com/vega/ipyvega

I think we can at least make the width and height optional so that users do not have to specify them when users use responsiveSize. But, yes, it is somewhat challenging to provide non-constant optimal width and height to individual tracks when the track/view structure becomes complicated.

Also, I was thinking that the appearance of some visual elements can be adaptive by default depending on the assigned size (e.g., hiding a color legend when the track height is too narrow). Another thing that we can figure out to make the responsiveSize more useful.

Originally posted by @sehilyi in #85 (comment)

Hi, I am trying to use gos to visualize a local bed file exactly like the OP in #120 and am also having trouble. I am working on a python notebook and the visualization is just a blank canvas.

My code:

import gosling as gos

data = gos.csv(
    url="./jw18_NC_002978.6_cov.bed",
    headerNames=['chrom', 'position', 'value'],
    chromosomeField="chrom",
    genomicFields=["position"],
    
    separator="\t",
)
gos.Track(data).mark_bar().encode(
    x=gos.X("position:G"),
    y=gos.Y("value:Q", axis="left"),
).view(
    assembly=[
        ("NC_002978.6", 1267782),
    ]
)

Head of the bed file:

NC_002978.6  0     0
NC_002978.6  3428  1
NC_002978.6  3439  2
NC_002978.6  3441  6
NC_002978.6  3446  8
NC_002978.6  3451  12
NC_002978.6  3454  16
NC_002978.6  3462  17
NC_002978.6  3463  21
NC_002978.6  3468  22

The bed file is only ~2 Mb (~100,000 lines) so I don't think file size is the issue here.

Screenshot for clarity on blank canvas:

Would appreciate any advice! Thank you.

The renderer for gos is very simple and doesn't need a Jupyter to work. We could create a widget that creates a "live" rendering of the View and exposes some gosling functionality:

import gosling as gos
from gosling.widget import GoslingWidget

view = gos.Track(...).mark_rect().encode(...).view()
view # simple renderer

widget = GoslingWidget(view) # takes a gosling root object
widget # live widget

Hi,

First, thanks for the great work! I am really liking the layout of Gosling so far!

I am having an issue plotting some data. I have a microbial assembly with two chromosomes and a tsv file with three columns: chromosome, position, value (the file does not have a header). I want to plot the values along each chromosome, so have code like this:

data = gos.csv(
    "../data/06_coverage/23b_coverage.bed",
    separator="\t",
    headerNames=["chrom", "position", "value"],
    chromosomeField="chrom",
    genomicFields=["position"]
)

gos.Track(data).mark_bar().encode(
    x=gos.X("position:G"),
    y=gos.Y("value:Q", axis="left"),
).view(
    assembly=[
        ("NC_012345.1", 1_234_567),
        ("NC_012346.1", 45000)
    ]
)

When I run the plot, I just get a blank canvas with the chromosome labels/positions along the top, but no data is showing up. I am not sure what I am missing here. Does the file have to have two coordinates for each feature?

Thanks in advance for the help! I hope this is not something obvious that I am missing.

Keep up the great work! Excited to see this tool continue to develop!

I am trying to use python gos to create specs that I can then plug into a react app. The react app uses gosling 0.9.30 and react ^18.2.0 and the python gos I am using is pip installed gosling-0.0.11. So far, I can create simple visualization that works perfectly within the app with this "hand made" spec:

  tracks: [
    {
      assembly: [["NC_010079.1", 2873728]],
      data: {
        url: "http://localhost:5000/gene",
        type: "csv",
        separator: "\t",
      },
      mark: "rect",
      x: { field: "start", type: "genomic" },
      color: { field: "start", type: "quantitative", legend: true },
      width: 600,
      height: 130,
    },
  ],
};

I can also create a visualization within jupyter notebook using gos and save it as a HTML file. But, when I plug in a spec created with gos into the react, I get all the components but none of the data is displayed. I tried upgrading the python gos to the latest version, but the exported spec from that causes the app to crash. So what is the best way to figure out compatible versions without trying all of them?

Do you by any chance know if a bigwig file can be used directly without any pre-aggregation using vector()? It looked so according to the HiGlass docs, but when I tested with a local bigwig file, it did not show any visual elements in the rendered track, having some error message.

v = vector(
    "../data/Astrocytes-insertions_bin100_RIPnorm.bw",
    value="value",
    column="position",
    binSize=4
)

gos.Track(v).mark_bar(outlineWidth=0).encode(
    x="position:G",
    y=gos.Channel("value:Q"),
    color=gos.Channel("value:Q", legend=True),
    stroke=gos.value("black"),
    strokeWidth=gos.value(0.3),
).view() # Uncaught (in promise) TypeError: Cannot read property 'chromsizes' of null

I wonder if chromsizes should be somehow provided before storing a bigwig file to the server.

If a coordSystem is specified for the bigWig, but no chromsizes are found on the server, the import will fail.

Originally posted by @sehilyi in #39 (comment)

Hi,
I'm trying to plot some data and having trouble loading these data from CSV files using URL. My visualization is generated nicely with local data, but I want to be able to share it. According to Gosling documentation I should be able to load my data through URL directly without an Higlass server. So, I made a few tests and it turn out that I don't succeed to load my data from my server, while I do when I load the data from Dropbox. I have no clue of why or how to handle it, and I haven't find any documentation of how Gosling load data from URL or if there are any specific requirements to do so. Here is an example of the data I'm trying to load. Maybe you have an idea of how to make it work, or you can enlighten me of how Gosling function?

Thanks a lot for your work developing this tool, and I hope you can help me with my issue.
Cheers

The site is just the minimal boilerplate to have a working example gallery. It desperately needs better navigation, layout, and styling.

We have the .save() function explained on the doc under API Reference, but it is quite hidden to users and does not provide detailed explanations and examples. I think we need to add a dedicated page or section that describes ways to export Gos plots (e.g., html, json). Maybe we can add a section near the Local Data?:

Related tweet: https://twitter.com/sminot/status/1559746056669126657?s=21&t=yCLH8vli7kbjDf8foZhy5g

Before Update:

gos.Track(data).mark_text().encode(...,
  visibility=[
      gos.VisibilityCondition(operation="LT", measure="width", threshold="|xe-x|", transitionPadding=30, target="mark"),
      gos.VisibilityCondition(operation="LT", measure="zoomLevel", threshold=10, target="track")
  ]
)

After Update:

gos.Track(data).mark_text().encode(...).visibility_condition(
   operation="LT", measure="width", threshold="|xe-x|", transitionPadding=30, target="mark")
).visibility_condition(
   operation="LT", measure="zoomLevel", threshold=10, target="track"
)

Potentially related to #80.

I am unable to reproduce (as the error has now gone away), but I ran into a CORs issue with Chrome 95 the first time I tried to load a local CSV. I switched to Firefox and the visualization worked fine. When I navigated back to Chrome, the CORs issue went away and hasn't come back.

I will add more information here as I learn more, but I wanted to file something to note the issue is known.

Motivation

At the moment, data transformations must be specified in the constructor or .properties method for gos.Track.

gos.Track(...).mark_bar().encode(
  # ...
).properties(
	dataTransform=[{ ... }, { ... }],
)

Similar to altair, we could add top-level utilities for chaining transformations as additional methods on gos.Track.

gos.Track(...).mark_bar().encode(
# ...
).transform_filter(...).transform_log(...) # internally appends to `dataTransform`

Initial work in #3. It might not be possible with how gosling renders, but ideally we could capture a static image of the initial rendering and insert that into the metadata for the jupyter notebook. That way, when notebooks are shared statically (GitHub/initial opening), there is a picture where the interative visualization will be.

see: https://github.com/vega/ipyvega/blob/master/notebooks/VegaLite.ipynb

Using data utilities requires explicit loading each function explicitly:

import gosling as gos
from gosling.data import ...
from gosling.experimental.data import ...

This is nice to be able to toggle between that data apis, but I think we should lift the data utils to the top-level so they can be accessed via the gos import.

import gosling as gos

data = gos.bigwig(...)

With #65, I think this could provide similar control of resources to end users. We can also avoid naming collisions with python modules like csv and json since our functions are attached to the gos namepace rather than bare imports with these names. Additionally, the pandas extension mimics the usage of gos which is pleasant:

import pandas as pd
import gosling as gos

df = read_csv('./data.csv')

data = gos.csv('./data.csv', ...)
data = df.gos.csv(...)

It is easiest to reuse a gos.Track instance for subsequent partial track definitions. This is fine, but it does lead to larger JSON that could be avoided. Similar to Altair, we could look into consolidated duplicate data in a parent Track if users use gos.stack or gos.overlay methods:

ref:

https://github.com/altair-viz/altair/blob/8ff06cfbbfb470aba6798999c61128be398c4996/altair/vegalite/v4/api.py#L40

In the gosling website, there is an example of setting up visibility based on zoomLevel instead of width|height. If I try to set the visibility by passing zoomLevel to the measure parameter, I get schema validation error.

seq_info = seq_track.mark_text(
).encode(
    text=gos.Text("seq:N"),
    color=gos.value("black"),
    stroke=gos.value("white"),
    strokeWidth=gos.value(3),
    x=gos.X("start:G", linkingId='detail-1'),
    xe="end:G",
    row=gos.value(80)
).visibility_le(
    target="mark",
    measure="zoomLevel",
    threshold=10000,
    transitionPadding=5,
)

The code above gives:

SchemaValidationError: Invalid specification

        gosling.schema.core.VisibilityCondition->0->measure, validating 'enum'

        'zoomLevel' is not one of ['width', 'height']

However, I can pass width as a measure which passes the schema validation, then manually edit the json to zoomLevel and it will give me the exact behavior I am expecting i.e. mark appears when the zoomLevel is at < 10,000 bp. It would be nice to be able to use the function to set measure as zoomLevel instead of setting it up manually.

Right now gos.Data is just an alias for gos.DataDeep. At a minimum, it would be nice to add a handful of data aliases:

import gosling as gos
from gosling.data import bigwig

data = bigwig("...", ..., ...) # returns gos.DataDeep(type="bigwig", url="...", ....)

It would be even more exciting to inspect the first argument of this function and if a local file, start some kind of background server, returning a data class with the server url.

import gosling as gos
from gosling.data import bigwig

data = bigwig('./path/to/data.bigwig', ...) # returns gos.DataDeep(type='bigwig', url='http://localhost:8080/data.bigwig', ...)
track = gos.Track(data).mark_bar().encode(...).properties(...)
track.view()

Hey,
Just learned about gosling and am trying to play around with it using data from bacteria. How can I pass coordinates from my own bacterial assembly? I tried passing assembly='unknown' to track, but nothing renders when I do that.

I am currently using gos on jupyter notebook.

It seems like this might be a common pattern that we could do automatically in the view concat methods.

gos.horizontal(track.view(), track2.view(), view)
# vs
gos.horizontal(track, track2, view) # auto-calleds track.view() internally

Originally posted by @manzt in #19 (comment)

Not that important for now, but we can edit error or warning messages to make it more understandable in the context of Gosling, if possible.

For example, "chart specification" in the following screenshot can be changed to "view specification," or more generally "gosling specification."

@manzt Is it possible to use pandas for documentation?

We can only use pandas in the docs if the data doesn't require a server at all (e.g. json data). This is very straightforward to implement:

diff --git a/gosling/__init__.py b/gosling/__init__.py
index a28b501..5d4b92c 100644
--- a/gosling/__init__.py
+++ b/gosling/__init__.py
@@ -16,3 +16,7 @@ class GosAccessor:
         content = self._df.to_csv(index=False) or ""
         url = data_server(content, extension="csv")
         return dict(type="csv", url=url, **kwargs)
+
+    def json(self, **kwargs):
+        values = self._df.to_dict(orient="records")
+        return dict(type="json", values=values, **kwargs)

But I have chosen to not implement this at the moment because currently the data would be embedded in the spec every time it is referenced. For this example, it would be just fine since each dataset are only referenced once, but it doesn't seem to work (I see the CSVDataFetcher ends up getting referenced and breaks).

diff --git a/gosling/examples/between_link_pandas.py b/gosling/examples/between_link_pandas.py
index 6dd1bfc..730254b 100644
--- a/gosling/examples/between_link_pandas.py
+++ b/gosling/examples/between_link_pandas.py
@@ -41,8 +41,8 @@ column_info = [
     {"chromosomeField": "first_chr", "genomicFields": ["first_start", "first_end"]},
     {"chromosomeField": "second_chr", "genomicFields": ["second_start", "second_end"]}
 ]
-data_bg = df_bg.gos.csv(genomicFieldsToConvert=column_info)
-data_hl = df_hl.gos.csv(genomicFieldsToConvert=column_info)
+data_bg = df_bg.gos.json(genomicFieldsToConvert=column_info)
+data_hl = df_hl.gos.json(genomicFieldsToConvert=column_info)

Originally posted by @manzt in #64 (comment)

Hi! I'm running this simple script to visualise bigwig file:

import gosling as gos
data = gos.bigwig(
    url=‘https://s3.amazonaws.com/gosling-lang.org/data/ExcitatoryNeurons-insertions_bin100_RIPnorm.bw’,
    column= ‘position’,
    value=‘peak’
)
track = gos.Track(data).mark_area().encode(
    x=gos.X(‘position:G’),
    y=gos.Y(“peak:Q”, axis=“right”),
    color=gos.value(“royalblue”),
).properties(title=“bigwig”,height=180,width=725)
vis = track.view(title=‘test’)
vis.save(‘test.html’)

if instead of your url I saved the file on my Amazon aws and run the same script using this url:
https://enrmil.s3.eu-north-1.amazonaws.com/ExcitatoryNeurons-insertions_bin100_RIPnorm.bw
I obtain an empty track. This is also happening if I use a localhost.

There appear to be temperamental issues with loading the JS to power the gos visualization in Jupyter Notebooks. It is difficult to debug due to the browser cache and (hidden) state of saved jupyter notebooks as well.

Steps to reproduce:

• open a new blank notebook (jupyter notebook) (Python 3.9, Chrome 95)
• load gos and execute a cell which renders a gos.View
• re-run the same cell

However, if a visualization is executed in a different cell, the expected rendering behavior is restored.

My best guess is that this is related to the custom JS loading code in gosling/display.py, but this is a requirement currently since some of our JS is incompatible with the global requirejs in Jupyter Notebooks.

TL;DR - If you are having trouble with rendering, try executing a different cell and re-running the previous cell. This step only needs to be done once, and will likely occur naturally during a typical workflow. Hopefully we will have a better solution soon.

vega-lite api is a good example of apis for creating vega-lite json specs. Gos is similar to vega-lite api, but for creating gosling json specs. I think it's more convenient for me to create a dataviz app with js apis rather than python apis because I use React to build the frontend for such kind of apps.

It looks like non-encoding-related properties (e.g., alignment, arrangement) are defined either inside properties(...) or directly in Track(...)/Chart(...):

gos.Track(...).encoding(...).properties(
   alignment="stack"
)

// vs

gos.Chart(
   arrangement="horizontal"
)

gos.Chart().properties(arrangement="horizontal") // error, no `Chart().properties()`

Perhaps, we can think about how to unify Track and Chart, and which options we want to support in each of properties(...), Track(...), and Chart(...).

Just learned of gosling today via twitter. I used to work at heavy.ai on large-scale dataviz and have been pretty close to the Vega/VegaLite ecosystem for a few years, including collaborations with that team.

First of all, wanted to say that this is a FANTASTIC project - kudos to everyone involved for a truly next-gen way of taking genomics data and turning it into a comprehensive set of visualizations. I've always felt that this area has been underserved and you've all done a tremendous job!

I work at sneller.io now - which may be of some interest. We allow for SQL on very large-scale JSON collections - see here. It may be of interest if you have data collections of this sort.

Next - One of the projects I worked on in the past is IbisData - This allows you to use SQL databases/DWs as backends in a pythonic data workflow without needing SQL directly. It might be interesting to explore how to back gosling/gos with Ibis so you can put large genomic data sets in central storage and use Ibis to only pull in the data needed on demand.

Hope this helps - again, fantastic work and will stay close to this!

I tested out gosling[all]==0.1.1 since I was going to update the tutorial.

Interestingly, scroll to zoom does not behave as expected in VSCode on my machine (OSX Sonoma, VSCode 1.84.2). It zooms slightly, but quickly starts scrolling the notebook. I also tried clicking the visualization before a scroll zooming and it had the same effect.

I tried out 0.1.1 in the Colab notebook and it worked fine, so it appears that it is environment specific.

If anyone has any hunches as to why this may be happening I'd love to hear and I can start poking around.

Just a matter of priorities. We have a separate repo/package from the Gosling Jupyter Widget because the build complexity was quite a bit. However, anywidget should make it feasible to move the widget to this repo and keep things together. This is what altair does now.

https://gosling-lang.github.io/gos/gallery/bar_chart.html there is no visuals on the website and locally running the code produced the same blank output. Please investigate.

I'm having trouble using gos to generate a heatmap of .cool data I have on my local device. I've tried specifying a local path from my machine's hard drive, a local path from my platform (ie uploading my data to Google Colab then running the code from there), as well as uploading my data to Google Drive, mounting Drive to Colab, and trying to specify a path from the mounted Drive. All of these result in the successful generation of an empty heatmap plot, but no data is visible.

That said, I am able to successfully generate a complete interactive heatmap using the test code and data supplied.

I'm wondering if maybe it's because I'm not specifying the data as a URL, as this is the only difference between the test code and my own code? Anyone have any thoughts?

Dear gos,
I am going to visualize my own data (BED, bedgraph or bigwig), my data format is very simple. But I can not find from your document how to visualized a user's data. Thank you!

Data format of bedgraph:
chr1 10468 10469 1.000
chr1 10469 10470 0.667
chr1 10470 10471 1.000

Thank you!
Yang

Since both Track and View are important concepts in Gosling, I think we can support explicit functions for both in the python package as well. In this regard, I was wondering whether it makes sense to rename Chart to View since Chart() looks to construct Gosling's views.

// from
gos.Chart(
    title='Overview and Detail Views',
    arrangement='horizontal',
    views=[overview, detail_view]
)

// to
gos.View(
    title='Overview and Detail Views',
    arrangement='horizontal',
    views=[overview, detail_view] // two children views that are nested in this parent view
)

So, at the root level, users define a View, and it can contain either multiple Views or Tracks.

It might be nice to add some convenience exports for reusable example datasets for gosling. This could remove some of the boilerplate in the examples for:

import gosling as gos
- from gosling.data import multivec
+ from gosling.datasets import cistrome_multivec


- data = multivec(
-    url="https://server.gosling-lang.org/api/v1/tileset_info/?d=cistrome-multivec",
-    row="sample",
-    column="position",
-    value="peak",
-    categories=["sample 1", "sample 2", "sample 3", "sample 4"],
-    binSize=5,
- )
- base_track = gos.Track(data, width=800, height=100)

+ base_track = gos.Track(cistrome_multivec, width=800, height=100)

There is another script in altair to autogenerate documentation from the generated source code. Perhaps we could use some inspiration from this as well:

https://github.com/altair-viz/altair/blob/master/tools/generate_api_docs.py