If a Comment or Submission's author is queried with self.resp.author.name or self.resp.author.id, reddit-detective throws an error and stops the network creation (or whatever operation is being carried).

Will be using Sonarcloud to track duplications, security vulnerabilities, and technical debt at the code level to test production readyness.

In network.py, a uniqueness constraint defined for comments is missing.

Input:

# necessary imports are done, praw integration works fine
# no problems on the DB side
net = RedditNetwork(
        driver=driver,
        components=[
            CommentsReplies(Submission(api, "li5sts", limit=5))
        ]
    )
# constraints are already created
net.run_cypher_code()

Output:

Traceback (most recent call last):
  File "C:/Users/Ümit/PycharmProjects/rdtest/aaa.py", line 32, in <module>
    net.run_cypher_code()
  File "C:\Users\Ümit\PycharmProjects\rdtest\venv\lib\site-packages\reddit_detective\network.py", line 95, in run_cypher_code
    self._run_query(codes=self._codes())
  File "C:\Users\Ümit\PycharmProjects\rdtest\venv\lib\site-packages\reddit_detective\network.py", line 83, in _codes
    codes = list(chain.from_iterable([point.code() for point in self.components]))
  File "C:\Users\Ümit\PycharmProjects\rdtest\venv\lib\site-packages\reddit_detective\network.py", line 83, in <listcomp>
    codes = list(chain.from_iterable([point.code() for point in self.components]))
  File "C:\Users\Ümit\PycharmProjects\rdtest\venv\lib\site-packages\reddit_detective\relationships.py", line 154, in code
    comment_merges, comment_links, submissions = self._merge_and_link_comments(self.comments())
  File "C:\Users\Ümit\PycharmProjects\rdtest\venv\lib\site-packages\reddit_detective\relationships.py", line 196, in comments
    comment.refresh()
AttributeError: 'MoreComments' object has no attribute 'refresh'

There are many components of reddit-detective which can and should be unit tested. Add unit tests for those stuff, separate them from e2e tests

Assume that K comments are under the same submission.
During the code generation, the code to create a node for the given submission is generated K times.

Shortcomings of the current solution:

• Forces the user to create a network with a degree and then generate the code, to get a code with no duplicates.

There might be instances where one wants to keep their interaction with Neo4j minimum, or one does not even use Neo4j. So, the analytics functions should work for NetworkX objects as well. (NetworkX is chosen since it is one of the most popular and easiest-to-use graph data science libraries)

Currently, the code is stored first and then executed. The code is held in a list of strings, with each element being one "component". To reduce space complexity, each "component" should be executed just after its creation

After fixing #22, add CI using GitHub Actions

On-the-spot scraping is very unscalable due to delays caused by Reddit API calls. RD should be usable by people who want to analyze some big data. Given any tabular data with some description from the user, we should be able to convert it to a Neo4j Graph.

Seems like the commits to reduce redundancy also led to incomplete graphs. The next version will involve a rollback, and please do not use v0.1.2. (I will also add this notice to the docs)

Problem

Currently, the main reason for high running times is the rate limit of the Reddit API. So we should minimize the number of calls to reduce running times.

Validation of the Problem

Test code:

# Assume api_ is declared and Neo4j is started
net = RedditNetwork(
        driver=driver_,
        components=[
            Comments(Redditor(api_, "Anub_Rekhan", limit=5))
        ]
    )
net.run_cypher_code()

The rate-limit evasion function in PRAW is called 59 times by the code above.

The code creates 18 nodes and 22 relationships atm. 59 is simply too much for such a small graph.

cProfile results:

Why are there so many requests?

TL;DR: The current version sends an unnecessary amount of requests.

Num of calls - Redditor object itself: 1

Num of calls - Getting the complete list of comments: 5 (2 items in the list of 11 comments are duplicate. That'll be fixed)

Num of calls - Merge and Link Comments: 33; At each iteration: 3 (1 Comment, 1 Author, 1 Submission). 3 * 11 = 33

Num of calls - Merge and Link Submissions: 22; At each iteration: 2 (1 Subreddit, 1 Author). 2 * 11 = 22 (There shouldn't be 11 submissions instead of 2. That'll be fixed)

The total number of calls is 61 (Same with the number in the Call Graph)

What to do?

• "Merge code" generation shouldn't require a data_models object. Maybe abstract? (Affects 55 calls in the example above)
• Find other ways of getting the complete list of comments (Affects 5 calls in the example above)

Due to calling self.start.submissions() multiple times, the relationship creation process is really slow.

Needed:

• YML files to integrate with readthedocs.io and MkDocs
• Documents for v0.1.1

Before moving on to development/code reviews, I'm going to create docs of the codebase for potential users and potential contributors.

Any tips are appreciated at this point, especially from experienced and/or detail-oriented people

I'll use readthedocs.io and build docs with MkDocs, using Markdown files

For v0.1.1, the docs will be organized in this way:

docs/
    index.md (will be the same with ../README.md)
    analytics/
        metrics.md
    data_models.md
    relationships.md
    network.md

Will create a markdown file including:

• Resources for people interested in contributing (PRAW documentation, Neo4j tutorials, introductory material to Social Network Analysis, etc.)
• Issue standards
• PR guidelines

ps: It's the first time I got an outside contribution to an open-source project of mine. Thank you, Lajos!

In the lines down there, reddit-detective gets rid of duplicate code fragments in the cache while preserving the order of elements in it. However, the Cypher code generated by method .code() still has duplicates.

https://github.com/umitkaanusta/reddit-detective/blob/main/reddit_detective/network.py#L79-L86

def _codes(self):
    """
    Get codes for every component
    """
    codes = list(chain.from_iterable([point.code() for point in self.components]))
    # Remove duplicates without changing order
    codes = sorted(set(codes), key=lambda x: codes.index(x))
    return codes

Given test case (the debugger shows the length of the list codes before and after the operation above):

# the necessary imports are done 
obj = CommentsReplies(Submission(api_, "jpt7s7", limit=None))
net = RedditNetwork(
    driver=driver_,
    components=[
        obj
    ]
)
net.run_cypher_code()

# 83 nodes
# 101 connections
# length of "codes": 332
# length of "codes" after the operation above: 184

The code used to get the duplicates:

# the continuation of the code above
# the variables obj and net are the ones created above
from pprint import pprint
from collections import Counter
L1 = obj.code()
L2 = list(net._codes())pprint(Counter(L1) - Counter(L2))

Duplicates:

Counter({'MERGE (:Submission {id: "jpt7s7", created_utc: 1604764778.0, title: "Play detective on Reddit: Discover political trolls, secret influencers and more", text: ""});': 49,
         '\nMATCH (n1 {id: "jpt7s7"})\nMATCH (n2 {id: "2qh0y"})\nWITH n1, n2\nMERGE ((n1)-[:UNDER {}]->(n2));\n': 49,
         '\nMATCH (n1 {id: "d1nfvrn"})\nMATCH (n2 {id: "jpt7s7"})\nWITH n1, n2\nMERGE ((n1)-[:AUTHORED {}]->(n2));\n': 49,
         'MERGE (:Redditor {id: "d1nfvrn", username: "Anub_Rekhan", created_utc: 1504794766.0, has_verified_email: "True"});': 1})