I receive an issue when trying to color nodes by a node attribute. Each node has an attribute labeled 'attr_1' and each attribute value is a string. The CircosPlot object returns a KeyError: 'attr_1'

@MridulS, moving this over from our email thread.

Seems like we've got something good going here. Perhaps we should somehow state clearly how others can use NAMS material for teaching purposes, something akin to how the Software/Data Carpentry organization allows the material to be used freely. I'm thinking of adding some verbiage to the README:

Teaching with this repository material

If you would like to teach with this repository material, we request only the following:

• Proper attribution back to the original in the primary landing page of derivative work. (This would usually imply the README.)
• Modifications to the original material documented in forked repository. (An example would be a plain text HISTORY file, or as a section in the forked README.)
• Loop back to us if there are recordings - so that we can link to them in the original repository!

The text is experimental, so if you're ok with the idea but want to propose a change, please do so!

Hi, I followed the instructions, install circos with pip install, checkenv.py failed with:

Traceback (most recent call last):
File "checkenv.py", line 19, in
'{0} not present. Please install via pip or conda.'.format(p)
AssertionError: circos not present. Please install via pip or conda.

Re-ran pip install circos and got:

Requirement already satisfied (use --upgrade to upgrade): circos in /home/jh/anaconda3/envs/network/lib/python3.5/site-packages

I've set up the conda environment with the packages listed on the github page, but when I go to launch the Jupyter notebook, I get:

(Sci2018PyNetworkAnalysis) eric@mercury ~/SciPy 2018 Materials/Network-Analysis-Made-Simple/Network-Analysis-Made-Simple $ jupyter notebook
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fad11a0b740 (most recent call first):
Aborted

This function uses manipulation of powers of the adjacency matrix (via numpy & scipy) to detect all of the potential triadic closures in a graph. The key fact to note is that the n-th power of the adjacency matrix corresponds to a matrix containing the number of n-hop walks between each pair of nodes. Some tests on various graphs are included below the function.

import networkx as nx
import numpy as np

from scipy.sparse import triu

%matplotlib inline

def get_all_open_triangles(G, nodes=None):
    """ Detect all potential triadic closures in a graph, using manipulation of powers of the adjacency matrix.

    G: Networkx undirected graph (of type networkx.classes.graph.Graph)
    nodes: Node list which, if passed, will be used to order the result

    returns: List of 2-tuples containing nodes which, if connected, would form a new triangle (triadic closure)
    """
    if type(G) is not nx.classes.graph.Graph:
        raise ValueError('This method only works for undirected graphs')

    # Use specified node order, if given
    nodes = G.nodes() if not nodes else nodes

    # Retrieve adjacency matrix and its square
    adj1 = nx.adjacency_matrix(G, nodes)
    adj2 = adj1**2

    # According to basic graph theory, the n-th power of the adjacency matrix
    # corresponds to a matrix containing the number of n-hop walks between 2 nodes
    # Note that we're only concerned with the upper triangle of these matrices,
    # since the graph is undirected and we also don't care about the diagonal
    one_hop = np.nonzero(triu(adj1, 1) > 0)
    two_hop = np.nonzero(triu(adj2, 1) > 0)
    one_hop_set = set(zip(one_hop[0], one_hop[1]))
    two_hop_set = set(zip(two_hop[0], two_hop[1]))

    # Take the difference of the two sets, and any positive entries represent open triangles
    # That is, nodes with a 2-hop path, but no 1-hop path, comprise an open triangle
    open_triangle_indices = two_hop_set.difference(one_hop_set)

    # Map the open triangle indices back to an interpretable list of nodes
    triangle_nodes = [(nodes[ix[0]], nodes[ix[1]]) for ix in open_triangle_indices]

    return triangle_nodes

# Example 1: Simple Graph
G = nx.Graph([('A', 'B'),
              ('A', 'D'),
              ('B', 'D'),
              ('D', 'C')])
nx.draw(G, with_labels=True)
print(get_all_open_triangles(G, ['A', 'B', 'C', 'D']))

# Example 2: Barabasi Albert Graph
bag = nx.barabasi_albert_graph(15, 2, 999)
nx.draw(bag, with_labels=True)
print(get_all_open_triangles(bag))

# Example 3: Caveman Graph
cmg = nx.caveman_graph(4, 3)
nx.draw(cmg, with_labels=True)
print(get_all_open_triangles(cmg))

# Example 4: Florentine Families Graph
ffg = nx.florentine_families_graph()
nx.draw(ffg, with_labels=True)
print(get_all_open_triangles(ffg))

# Example 5: Newman Watts Strogatz Graph
nwsg = nx.newman_watts_strogatz_graph(10, 2, .5, 999)
nx.draw(nwsg, with_labels=True)
print(get_all_open_triangles(nwsg))

Given the sociopatterns network, if an actual disease were to spread across the network, can you simulate how it would go?

Pre-requisite knowledge:

• Linear algebra (for quick simulation)

Should probably be a "project" notebook/case study notebook at the end.

As per title.

It should be inserted into NB2.

I think it might be a good idea to add in an exercise that involves finding a particular node, or a particular edge.

I am struggling to install the nxviz package with pip. It gets as far as fulltoc and stops with the following message:

Collecting sphinxcontrib-fulltoc==1.2.0 (from nxviz)
Downloading https://files.pythonhosted.org/packages/8e/a6/d1297db9b75650681e5429e92e13df139ee6b64303ff1b2eea4ebd32c0a9/sphinxcontrib-fulltoc-1.2.0.tar.gz
Complete output from command python setup.py egg_info:
Download error on https://pypi.org/simple/pbr/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
Couldn't find index page for 'pbr' (maybe misspelled?)
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
No local packages or working download links found for pbr
Traceback (most recent call last):
...

I have a similar problem when I try install from a local archive downloaded from pipit.org. Is there some solution I could try?

Use while instead of for to avoid updating object being used as target of iterator.
Use generator in extend to avoid materializing list.

def path_exists(node1, node2, G):
    """
    This function checks whether a path exists between two nodes (node1, 
    node2) in graph G.
    
    Special thanks to @ghirlekar for suggesting that we keep track of the 
    "visited nodes" to prevent infinite loops from happening.
    
    Reference: https://github.com/ericmjl/Network-Analysis-Made-Simple/issues/3
    
    With thanks to @joshporter1 for the second bug fix. Originally there was 
    an extraneous "if" statement that guaranteed that the "False" case would 
    never be returned - because stack never changes in shape. Discovered at 
    PyCon 2017.
    
    With thanks to @chendaniely for pointing out the extraneous "break".
    """
    visited_nodes = set()
    stack = [node1]
    
    while len(stack):
        node = stack.pop()
        neighbors = list(G.neighbors(node))
        if node2 in neighbors:
            print('Path exists between nodes {0} and {1}'.format(node1, node2))
            return True
        else:
            visited_nodes.add(node)
            stack.extend(n for n in neighbors if n not in visited_nodes)
    
    print('Path does not exist between nodes {0} and {1}'.format(node1, node2))
    return False

Hi!

First, thank you for making this cool resource openly available. I cloned the repo, set up the nams conda env and ran setup.py. However, in the second notebook lesson (2-networkx-basics-student.ipynb), it fails to import custom in the top cell. Same thing when trying on binder. Am I missing a setup step still?

In the notebook 2-networkx-basics-instructor.ipynb, the first code cell after the Basic Network Statistics section

# Who are represented in the network?
G.nodes()[0:5]

returns the following error on my Macbook:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-3fb6155cc2e4> in <module>()
      1 # Who are represented in the network?
----> 2 G.nodes()[0:5]

~/playspace/pycon2018/Network-Analysis-Made-Simple/network/lib/python3.6/site-packages/networkx/classes/reportviews.py in __getitem__(self, n)
    176 
    177     def __getitem__(self, n):
--> 178         return self._nodes[n]
    179 
    180     # Set methods

TypeError: unhashable type: 'slice'

I can fix the issue with

list(G.nodes())[0:5]

Let me know if this is an actual issue or just something not setup correctly on my end. Thanks!

$ pipenv install --skip-lock -r requirements.txt
$ pipenv install --skip-lock jupyterlab  

I was seeing some package errors if not using --skip-lock ...
i.e. Could not find a version that matches matplotlib==2.2.2,==3.0.1,>=2.0.0

And to create a env kernel named "network-env":

$ pipenv run python -m ipykernel install --user --name=network-env

@MridulS can you please build a conda-forge package for python-louvain? Its presence as a pip-installed package is causing some issues.

@MridulS this is non-urgent, but in case you have the time, I think notebook 8 has one line that is incompatible with NetworkX 2.4.

Specifically, in the latest PR #133, I changed the use of strongly_connected_component_subgraphs to:

pass_2015_strong = max(nx.strongly_connected_components_recursive(pass_2015), key=len)

This, however, causes the following code cell to fail:

nx.average_shortest_path_length(pass_2015_strong)

Do you think you could take a look at this one please? And once you're done with it, add back the notebook to our Travis CI checks?

  - jupyter nbconvert --config nbconvert_config.py --execute 8-US-airports-case-study-instructor.ipynb

Lemme know if you have any issues here.

It may help with the learning.

Suggest to start with NB4, it has the most complicated examples.

The most important one is cleaning notebook outputs.

Greetings! I am just having what appears to be a minor issue. I do want to say thank you for the ODSC West training you conducted, I really appreciated the instruction and materials. I wanted to use my own Anaconda instance for the notebooks, pretty sure that has nothing do with the problem, but wanted to mention just in case. I did use the prescribed yml file from the repo to create the env in Anaconda3.

Was running the "networkx-basics" notebook 2 (student version) and ran into the same error, screenshots below, when trying to display the ArcPlot and the CircosPlot.

Thank you in advance for your help! Let me know if you need further information.

-Jason

I tried cloning the git repository into anaconda on the terminal line but the first few times I tried it, it took forever to load and terminated before it finished due to an error. The error was "fatal: early EOFs: 82% (850/1036), 66.21 MiB | 28.00 Kib/s \n fatal: The remote end hung up unexpectedly \n fatal: index-pack failed \n error: RPC failed; curl 56 SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054"

I get error messages from anaconda when attempting to install hiveplot in environments with alternatively python 3.4 and 3.5. Clean install with python 2.7, however. Requirement of python 3 perhaps incorrect?

If I use environment.yml, I get a pandas.tslib error. Manually installing pandas via conda in a new environment doesn't give this error, but manually removing pandas and re-installing it in the original "network" environment does.

Should be:

def test_graph_integrity(G):
    assert 30 in G.nodes()
    assert 31 in G.nodes()
    assert G.nodes[30]['gender'] == 'male'
    assert G.nodes[31]['gender'] == 'female'
    assert G.has_edge(30, 31)
    assert G.has_edge(30, 7)
    assert G.has_edge(31, 7)
    assert G.edges[30, 7]['count'] == 3
    assert G.edges[7, 30]['count'] == 3
    assert G.edges[31, 7]['count'] == 3
    assert G.edges[7, 31]['count'] == 3
    assert G.edges[30, 31]['count'] == 3
    assert G.edges[31, 30]['count'] == 3
    print('All tests passed.')
    
test_graph_integrity(G)

In notebook 4:

Is:

# Compare for yourself that those are the only triangles that node 3 is involved in.
neighbors3 = G.neighbors(3)
neighbors3.append(3)
nx.draw(G.subgraph(neighbors3), with_labels=True)

Should be:

# Compare for yourself that those are the only triangles that node 3 is involved in.
neighbors3 = list(G.neighbors(3))
neighbors3.append(3)
nx.draw(G.subgraph(neighbors3), with_labels=True)

see summary

No status report for environment check, that is after running $ python checkenv.py

Queue size is exponentially growing and unbounded since same nodes are added in queue multiple times without checking if they have been traversed before. I think the algorithm also gets trapped in cycles, but I have not tested that.

I solved the issue by adding a list of visited nodes and maintaining a unexplored queue with unique elements and exclusive from visited nodes. Here's my code.

def path_exists(node1, node2, G, debug=False):
    """
    This function checks whether a path exists between two nodes (node1, node2) in graph G.
    """
    # Fill in code below
    queue = G.neighbors(node1)
    visited_nodes = []
    destination = node2

    for node in queue:
        visited_nodes.append(node)
        if(debug): print(visited_nodes, '\t', queue)
        neighbors = G.neighbors(node)
        queue.remove(node)
        [queue.append(neighbor) for neighbor in neighbors if neighbor not in visited_nodes and neighbor not in queue]
        if node2 in neighbors:
            print('Path exists between nodes {0} and {1}'.format(node1, node2))
            return True

    print('Path does not exist between nodes {0} and {1}'.format(node1, node2))
    return False

I also made a small graph as a test environment with a single cycle and a single isolated node.

G1 = nx.Graph()
G1.add_nodes_from([1,2,3,4])
G1.add_edges_from([(1,2),(2,3),(1,3)])
nx.draw(G1)

And a couple test cases

path_exists(2,4, G1, debug=True)
path_exists(1,3, G1, debug=True)

Gets me the output

[1]      [1, 3]
[1, 2]   [3, 2]
Path does not exist between nodes 2 and 4

[2]      [2, 3]
Path exists between nodes 1 and 3

By looking over the debug output, I think it solves the issue by conclusively saying there is no path once it has traversed each node in the graph.

Let me know if you find this helpful.

Hi, I'm trying to install the packages required for this tutorial and I'm running into a problem. Cirocos only likes Python 3.5 and refuses to conda install if I'm using Python 3.4

scipy-2016-tutorial brian$ conda install -c ericmjl circos
...
The following specifications were found to be in conflict:

• circos
• python 3.4*

But hiveplot only likes Python 3.4 and won't install in my Python 3.5 environment

scipy-2016-tutorial brian$ conda install -c ericmjl hiveplot
...
The following specifications were found to be in conflict:

• hiveplot
• python 3.5*

Any suggestions?

Exercise¶
Can you create a ranked list of the importance of each individual, based on the number of neighbors they have? (3 min.)

Hint: One suggested output would be a list of tuples, where the first element in each tuple is the node ID (an integer number), and the second element is the number of neighbors that it has.

Hint: Python's sorted(iterable, key=lambda x:...., reverse=True) function may be of help here.

In 2 lines with minimal materialization b/c uses a generator expression.

gen = ((len(list(G.neighbors(x))), x) for x in G.nodes())
sorted(gen, reverse=True)

on two separate tutorials, reception has been weak, likely because of its positioning towards the end of the tutorial, and because it requires quite a lot of foundation before it can work.

may want to demote this notebook to an "appendix"-style notebook for reference, but not actually taught during the tutorial session.

As per title.

Should be:

def load_sociopatterns_network():
    # Read the edge list

    df = pd.read_csv(
        'datasets/sociopatterns-infectious/out.sociopatterns-infectious',
        sep=' ', skiprows=2, header=None)
    df = df[[0, 1, 2]]
    df.columns = ['person1', 'person2', 'weight']

    G = nx.Graph()
    for row in df.iterrows():
        p1 = row[1]['person1']
        p2 = row[1]['person2']
        if G.has_edge(p1, p2):
            G.edges[p1, p2]['weight'] += 1
        else:
            G.add_edge(p1, p2, weight=1)

    for n in sorted(G.nodes()):
        G.nodes[n]['order'] = float(n)

    return G

Hey Eric - I got tripped up first few minutes of the class because I didn't realize how to run a notebook - would be good as part of the class setup or first few slides to have a test and instruction for running the notebook and make sure everything is OK.

Hi,
i want to work with Circos Plots and when i gave " from nxviz.plots import CircosPlot" I am getting this error ModuleNotFoundError: No module named 'nxviz' .
To add it I gave "! conda install -c conda-forge nxviz" which is giving me error

**Collecting package metadata (current_repodata.json): ...working... failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/conda-forge/win-64/current_repodata.json
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /conda-forge/win-64/current_repodata.json (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000000662A280048>, 'Connection to conda.anaconda.org timed out. (connect timeout=9.15)'))"))**

NOTE : I am working with spyder in Anaconda. ..all the commands are given in IPython Console.

Can someone please help me out with this issue ..??

Thanks in Advance..!

The 10th cell has nx.pagerank(passenger_graph). It returns an error:

---------------------------------------------------------------------------
NetworkXNotImplemented                    Traceback (most recent call last)
<ipython-input-10-903bde57db38> in <module>()
----> 1 nx.pagerank(passenger_graph)

<decorator-gen-326> in pagerank(G, alpha, personalization, max_iter, tol, nstart, weight, dangling)

~/anaconda/lib/python3.6/site-packages/networkx/utils/decorators.py in _not_implemented_for(not_implement_for_func, *args, **kwargs)
     69         if match:
     70             msg = 'not implemented for %s type' % ' '.join(graph_types)
---> 71             raise nx.NetworkXNotImplemented(msg)
     72         else:
     73             return not_implement_for_func(*args, **kwargs)

NetworkXNotImplemented: not implemented for multigraph type

cc: @MridulS, please fix this for execution purposes.

In order for checkenv.py to pass, ffmpeg must be installed. It is not included in the environment dependencies.

Using pew to create my virtual environment.

I had to install nxviz first to avoid these warnings

nxviz 0.3.6 has requirement matplotlib==2.1.2, but you'll have matplotlib 2.2.2 which is incompatible.
nxviz 0.3.6 has requirement networkx==2.0, but you'll have networkx 2.1 which is incompatible.
nxviz 0.3.6 has requirement numpy==1.14.0, but you'll have numpy 1.14.3 which is incompatible.

I did this by doing the following

$ pew new network
(network) $ pip install nxviz
(network) $ pip install -r requirements.txt

I don't know if this is an accepted solution, the notebooks seems to work, though.

I also had a little issue with the checkenv.py file. I had to change python-louvain to community to get it to pass my environment.

diff --git a/checkenv.py b/checkenv.py
index 24d7f30..b2757c4 100644
--- a/checkenv.py
+++ b/checkenv.py
@@ -10,7 +10,7 @@ def check_import(packagename):
         return False

 packages = ['networkx', 'numpy', 'matplotlib', 'hiveplot', 'pandas',
-            'jupyter', 'nxviz', 'python-louvain']
+            'jupyter', 'nxviz', 'community']

 all_passed = True

If I did something that will bite me in training I'd appreciate a heads up.

--- nxviz_fifth 2018-05-05 12:19:21.203861367 +0200
+++ nxviz_first 2018-05-05 12:17:57.912863465 +0200
@@ -23,15 +23,14 @@
 jupyter-client==5.2.3
 jupyter-console==5.2.0
 jupyter-core==4.4.0
-kiwisolver==1.0.1
 MarkupSafe==1.0
-matplotlib==2.2.2
+matplotlib==2.1.2
 mistune==0.8.3
 nbconvert==5.3.1
 nbformat==4.4.0
-networkx==2.1
+networkx==2.0
 notebook==5.4.1
-numpy==1.14.3
+numpy==1.14.0
 nxviz==0.3.6
 palettable==3.1.0
 pandas==0.22.0

As you can see the three packages mentioned above in the warnings are no longer the current version and I also seem to be missing kiwisolver for some reason when installing nxviz first.

The scatter plot for degree centrality vs number of neighbors gives a nice straight line with a positive slope when using the following code (same as the one from the instructor notes) -

neighbors = [len(G.neighbors(node)) for node in G.nodes()]

When using the code below, it gives a very wonky graph.
number_of_neighbors = [len(item[1]) for item in sorted(node_and_connections, key=lambda x: len(x[1]), reverse=True)]

May want to emphasize this point in the presentation.

Suggestion by Hannah after tutorial.

Not a huge deal but may prevent some confusion.

hi, i tried to follow instructions from readme - and the conda env create was failing with "cannot find package"

so i removed the matplotlib version number
and then finally python.app package

then i could create the environemtn, but was getting 'unknown symbol' during 'import hiveplot'

after bit of head scratching, found the solution

sudo apt-get install libqt4-dev
conda install sip
conda install pyqt

I think it might be a good idea to test the instructions in a fresh docker container, environments have probably changed (are changing)

In Network-Analysis-Made-Simple/3. Hubs and Paths (Instructor).ipynb

links = ((n1, n2) for n1, n2, d in G.edges(data=True))
reverse_links = ((n2, n1) for n1, n2, d in G.edges(data=True))

list(set(links) - set(reverse_links))

This is a simple fix.

Instead of:

from nams.solutions import some_module
from inspect import getsource

print(getsource(some_module))

We do:

from nams.solutions import some_module

some_module??

This will render the outputs with colouring done right (at least in the notebooks)!

That said, I could be wrong, and it might instead break the LeanPub outputs (though I've never tried).

I'd like to upgrade the repository a bit here, and make it into a format that is simultaneously compatible with:

• Running a tutorial
• Publishing an online book
• Exporting as a collection of PDFs to be printed.

This will be done in a branch. I'll be running some experiments on my personal static site server to see how this can be done most effectively.

I was having problems running the first cell of the notebooks due to a problem importing matplotlib

Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework

I solved this by creating a ~/.matplotlib/matplotlibrc file containing the line backend: TkAgg. Restarting jupyter notebook was also required.

[(n1, n2) for n1, n2 in G.edges() if not G.has_edge(n2, n1)]