shurcool / home Goto Github PK

The activity feed on the index page of dmitri.shuralyov.com shows events, such as:

Right now, the following 2 pieces of information are displayed separately:

1. the repository it happened in ("github.com/golang/go" in the screenshot above)
2. the title of the issue or change ("x/build/cmd/coordinator: fails ..." in the screenshot above)

I'm working primarily with Go, and the Go project has a well established convention for including the package as a prefix before the title, e.g., "net/http: add StatusTooEarly (425)". (It makes sites like goissues.org possible.) Additionally, many Go packages have vanity import paths that differ from the repository path that they're contained in, which we can find out relatively easily from go.mod files (see commit shurcooL/events@5bd98dd).

It's possible to use that information to compute the full import path of the affected Go package. Displaying that will be more readable and streamlined.

Screenshots

Example 1

For example, instead of seeing the repo and prefix on separate lines:

We can combine them and show:

Example 2

Instead of having to read "proxy:" before seeing that it's the x/net subrepo:

We can combine into a more readable "golang.org/x/net/proxy" import path:

Example 3

Instead of:

We can also include a snippet from the issue description body:

Example 4

This applies to third-party Go packages too. Instead of:

We can show the correct custom import path of that project:

Some commits can include generated files that are quite large (several MB). Current diff page will take a while to load, leading to a bad user experience. Especially when just trying to get a quick look (to see the commit message, files changed, etc.).

One step to improve this would be to avoid loading such files (either generated, or large) right away, but give the option to expand it later. That way the entire page loads quickly, and if a user really wants to see a large file, they can do so after explicitly clicking it.

For example, see second link in the commit message of commit 2536fa2.

Long titles overflow in an unexpected way in the activity stream. For example:

They should be forced to fit in one line, no matter the length. Ellipsis can be used to make it visible that the title is shortened (the tooltip is available to display the full title):

Thanks @katiehockman for noticing and reporting.

I'd like to move home from github.com/shurcooL/home to dmitri.shuralyov.com/website/home.

This is the issue tracking that task.

It may require resolving some issues related to dealing with large repositories, such as #15 and #17.

Existing issues and PRs (both open and closed) should be preserved in the move as well.

Facing this issue while running go get:

 go: github.com/spf13/[email protected] requires
	github.com/bketelsen/[email protected] requires
	cloud.google.com/go/[email protected] requires
	golang.org/x/[email protected] requires
	dmitri.shuralyov.com/gpu/[email protected]: unrecognized import path "dmitri.shuralyov.com/gpu/mtl": https fetch: Get "https://dmitri.shuralyov.com/gpu/mtl?go-get=1": dial tcp 172.93.50.41:443: i/o timeout

I am not able to access https://dmitri.shuralyov.com/gpu/mtl?go-get=1 from my local system either. Is it down?

Hi the link (http://dmitri.shuralyov.com/gpu/mtl) is not accessible right now and is causing build issue --
'''
go: dmitri.shuralyov.com/gpu/[email protected]: unrecognized import path "dmitri.shuralyov.com/gpu/mtl" (https fetch: Get https://dmitri.shuralyov.com/gpu/mtl?go-get=1: dial tcp 172.93.50.41:443: i/o timeout)
'''

As an example, when GitHub is experiencing availability issues, it should still be possible to view and receive notifications from Gerrit, and vice versa.

This was the only output when trying to view notifications during a GitHub outage this week:

did not get acceptable status code: 500 Internal Server Error body: "500 Internal Server Error\n\nGet \"https://api.github.com/notifications?per_page=1\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\n"

The current error checking is too strict, and bails out at the first error. Additionally, errors aren't displayed in the frontend. This can be improved.

If the server that is hosting the https://dmitri.shuralyov.com website is unavailable due to an outage of its current server provider (as has happened recently in #37), it may be worth considering a fallback plan.

It can be a quick-to-start second instance, or perhaps a more static page that makes use of the Go module versions already cached by https://proxy.golang.org and serving those instead.

This is the tracking issue to investigate and implement this.

Note that as more and more people update to Go 1.13 or newer, which uses https://proxy.golang.org by default, the benefit from doing this will diminish.

Going to the nonexistent URL https://dmitri.shuralyov.com/blog/1822 yields an HTTP 200 (should be 404) and the following non-nice error page:

template: issue.html.tmpl:13:13: executing "issue" at <.Issue>: error calling Issue: open /home/dmitri/Dropbox/Store/issues/dmitri.shuralyov.com/blog/issues/1822/0: no such file or directory

When someone merges a commit to the main branch of a repository on https://dmitri.shuralyov.com, its commit message should be scanned for any "Fixes" lines, and said issues should be closed. (See dmitri.shuralyov.com/website/gido/...@5bb3cef2 for a recent example where this would've been helpful.)

This requires:

• coming up with a well-defined syntax for the "Fixes" lines
  • it needs to be compatible with indieweb URLs
• implementing code to parse commit message and detecting "Fixes" lines
• performing an issue service API call to close said issue

Once implemented, similar functionality can be used to help resolve golang/go#29599.

Fetching https://dmitri.shuralyov.com/gpu/mtl?go-get=1
https fetch failed: Get https://dmitri.shuralyov.com/gpu/mtl?go-get=1: dial tcp 172.93.50.41:443: connect: connection refused

Hi, I wonder if this can be fixed again. I see you already faced this once. I guess our servers are in Japan.

Thanks!

This is what a (force) push that contains no new commits looks like:

There's extra whitespace at the bottom, making it uneven. Also, displaying an empty event containing a list of zero commits like that may not make much sense. Consider displaying the actual branch pushed to, the before and after commits?

These are the relevant public GitHub events for reference:

{
  "id": "6540416855",
  "type": "PushEvent",
  "actor": {
    "id": 1924134,
    "login": "shurcooL",
    "display_login": "shurcooL",
    "gravatar_id": "",
    "url": "https://api.github.com/users/shurcooL",
    "avatar_url": "https://avatars.githubusercontent.com/u/1924134?"
  },
  "repo": {
    "id": 90435863,
    "name": "shurcooL/events",
    "url": "https://api.github.com/repos/shurcooL/events"
  },
  "payload": {
    "push_id": 1958853295,
    "size": 2,
    "distinct_size": 2,
    "ref": "refs/heads/master",
    "head": "2a50f0a9442c7e233d4a6546087a0272224c116a",
    "before": "72b393d7a090ee0dfad31487bed00874dc13d9fa",
    "commits": [
      {
        "sha": "19caae42bd8ee707f6814a7a8a7535f0b4a8bdc4",
        "author": {
          "email": "[email protected]",
          "name": "Dmitri Shuralyov"
        },
        "message": "githubapi: Perform style changes.\n\nThe purpose of this commit is to make the diff of next commit smaller.\nIt applies various style changes to try to make the code more readable\nand consistent.\n\nThe two if err != nil statements after rc, err := s.fetchCommit are\nmerged into a single if/else if statement, to make it more clear that\nthe second if err != nil statement deals with the same error returned\nfrom fetchCommit. Rename the *github.ErrorResponse variable to e to\navoid shadowing original error.",
        "distinct": true,
        "url": "https://api.github.com/repos/shurcooL/events/commits/19caae42bd8ee707f6814a7a8a7535f0b4a8bdc4"
      },
      {
        "sha": "2a50f0a9442c7e233d4a6546087a0272224c116a",
        "author": {
          "email": "[email protected]",
          "name": "Dmitri Shuralyov"
        },
        "message": "githubapi: Detect \"merged\" state for comments on PRs.\n\nSince the IssueCommentEvent itself doesn't contain enough information\nfor us to know whether the PR has been merged or not, we fetch\nmentioned PRs (whether they've been merged). This is similar to us\nfetching additional information about mentioned commits.\n\nThen, we use the heuristic that if the PR is merged at present time,\nand the event had \"closed\" state, then it's replaced by \"merged\" state\ninstead.\n\nThis has a small chance of false positives. Imagine a PR was closed,\nthen someone commented on it, then it was reopened and merged. So at\npresent time, the PR is merged. But when the comment was made, the PR\nwas in fact a closed PR, not a merged PR.\n\nFixing this false positive would require fetching the time at which the\nPR was closed and comparing the event time to that. This should be done\nwhen there's an accessible test case for it, because it's hard\notherwise. Also, other parts of the code might have the same flaw, and\nshould also be vetted at that time.\n\nWhen we're using GraphQL API for fetching events, it'll likely be\nviable and trivial to directly query whether the PR was merged at the\ntime of event.\n\nResolves the following TODO comment:\n\n\t// TODO: Detect \"merged\" state somehow? It's likely going to require making an API call.",
        "distinct": true,
        "url": "https://api.github.com/repos/shurcooL/events/commits/2a50f0a9442c7e233d4a6546087a0272224c116a"
      }
    ]
  },
  "public": true,
  "created_at": "2017-09-01T15:21:20Z"
},
{
  "id": "6540330622",
  "type": "PushEvent",
  "actor": {
    "id": 1924134,
    "login": "shurcooL",
    "display_login": "shurcooL",
    "gravatar_id": "",
    "url": "https://api.github.com/users/shurcooL",
    "avatar_url": "https://avatars.githubusercontent.com/u/1924134?"
  },
  "repo": {
    "id": 90435863,
    "name": "shurcooL/events",
    "url": "https://api.github.com/repos/shurcooL/events"
  },
  "payload": {
    "push_id": 1958808559,
    "size": 0,
    "distinct_size": 0,
    "ref": "refs/heads/master",
    "head": "72b393d7a090ee0dfad31487bed00874dc13d9fa",
    "before": "2b2dabd52a5a682cbe7f4f0e35aed5d8225c5990",
    "commits": [

    ]
  },
  "public": true,
  "created_at": "2017-09-01T15:02:13Z"
},

The following error was observed on 2021-02-02 11:15:

GitHub Activity Service: walkSegMail(github.fileSeg{file:"/2021-02.reclog", skip:0, size:4096}): truncated record at offset 0: EOF
Gerrit Activity Service: ok

Restarting fixed the issue.

It's looks like a possible race condition: the file length was determined to be 4096 bytes yet reading at offset 0 returned EOF. Maybe the file was still being written after being created, and the buffers weren't flushed?

I want to move in the general direction of having web app logic run primarily on the frontend rather than on the backend. This means the web apps for notifications, issues, and changes (perhaps more in the future) should be implemented as a single-page application.

The motivation is to prioritize and improve:

• the speed at which I can iterate and implement changes to said applications
• ability to have highly interactive pages with real-time updates

At some short- and medium-term (but not long-term) cost to:

• page load performance
• page load resource size

(I might explain the reasons in more detail later, when I'm feeling less lazy.)

This is the tracking issue for this work.

When I comment (or do other actions) on local issue trackers (e.g., https://dmitri.shuralyov.com/issues/github.com/shurcooL/issuesapp/...$14#comment-1), corresponding events should appear in my personal activity feed. Currently, only events originating from GitHub are shown.

The IndieAuth specification has significant changes in 2020.
This implementation needs to be updated accordingly.

A very helpful blog post by @aaronpk covers these changes:

https://aaronparecki.com/2020/12/03/1/indieauth-2020

(Thanks to @vilhalmer for pointing it out to me.)

Here are results of prototyping this experience.

The hello repository did not exist in advance, it got created as I pushed:

Here are the corresponding events:

The "created repository" events become less relevant, and it's not really possible to provide a description. Might want to move towards "created package" events by discovering which Go packages were added as part of a push:

Need to think this through and ensure the end result would provide a coherent experience before committing.

Now that some repositories have a longer history, adding pagination to the history page is going to be a higher priority.

E.g., see https://dmitri.shuralyov.com/service/change/...$history:

This is a beginning...

A prototype is available at https://dmitri.shuralyov.com/projects/home-dark/.

References:

• https://drafts.csswg.org/mediaqueries-5/#prefers-color-scheme
• https://paulmillr.com/posts/using-dark-mode-in-css/ (/cc @paulmillr)

Our builds are broken with this error:

go: finding dmitri.shuralyov.com/service/change v0.0.0-20190320022222-9a7a6e8aef50
go: dmitri.shuralyov.com/service/[email protected]: git -c protocol.version=0 fetch --unshallow -f https://dmitri.shuralyov.com/service/change refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/ydnar/go/pkg/mod/cache/vcs/6337fe80c0c5be37685948ddcad19803db757c3ff59a7275f1d6cbb32466323d: exit status 128:
	fatal: the remote end hung up unexpectedly
go: error loading module requirements

Did you move your modules off GitHub?

Given that home implements a custom git server and module server, it is both helpful and a novel opportunity to perform some server-side validation on git commits being pushed, to ensure they produce valid Go module zips.

This can be useful to catch unintentional mistakes that a human can make, such as:

• accidentally including two file paths that are equal under Unicode case-folding (e.g., testdata/Data.txt and testdata/DATA.txt)
• accidentally adding a go.mod file that isn't all lower-case (e.g., Go.mod)
• accidentally having a mismatch between major version suffix (e.g., /v2) and the major version (e.g., v3.0.0)
• accidentally exceeding hard limits on file sizes that are enforced by the go command
• accidentally adding symbolic links or other irregular files that will not be included in the module zip
• accidentally adding a go.mod directory rather than file
• accidentally pushing commits with committer times that don't sort in strictly chronological order (e.g., after rebasing many commits, they all end up having equal committer times, which can sort badly)

When implemented, it means that doing a git push of a commit that would result in a bad Go module zip (one that the go command would reject, or one that can't include all the files that are in the commit) would be rejected by the git server by default (with an option to override it). It may look like this:

Now that the golang.org/x/mod/zip package exists, we can use its API to accomplish most of this. It can be used inside a server-side pre-receive git hook.

Issue #35 was about adding a server-side pre-receive git hook to reject certain commits. It works.

However, the logic in gitHandler.serveGitReceivePack to log events isn't aware of commits that were rejected, and ends up logging a false push event that never happened.

This can be fixed by changing the current code to somehow detect when the pre-receive git hook rejected the push, or alternatively, by factoring out the logic to log pushes into another git hook. That would allow removing the github.com/AaronO/go-git-http dependency.

Same as #23, I'm getting build errors when we attempt to download packages from your site:

go: dmitri.shuralyov.com/app/[email protected]: unrecognized import path "dmitri.shuralyov.com/app/changes" (https fetch: Get https://dmitri.shuralyov.com/app/changes?go-get=1: dial tcp 172.93.50.41:443: connect: no route to host)

curl -v https://dmitri.shuralyov.com
* Rebuilt URL to: https://dmitri.shuralyov.com/
*   Trying 172.93.50.41...
* TCP_NODELAY set
* connect to 172.93.50.41 port 443 failed: No route to host
* Failed to connect to dmitri.shuralyov.com port 443: No route to host
* Closing connection 0
curl: (7) Failed to connect to dmitri.shuralyov.com port 443: No route to host

I've encountered a strange bug due to which I can't authenticate to your site:

Previously my authentication attempts were successful when I used indielogin.com server, I even leave some reactions on your publications. But recently I am using my own server (source). Maybe the problem is related to this, but I'm not sure.

The Golang microformats2 parser shows that my photo is a string in an slice, so I don't really see how a map could be involved, unless you are using a your own parser that interprets the data in a different way:

{
  "items": [
    ...
    {
      "type": [
        "h-card"
      ],
      "properties": {
        "email": [
          "mailto:[email protected]"
        ],
        "family-name": [
          "Lebedev"
        ],
        "given-name": [
          "Maxim"
        ],
        "name": [
          "Maxim Lebedev"
        ],
        "nickname": [
          "toby3d"
        ],
        "note": [
          "Creative dude from Russia. Altruist, programmer and supporter of IndieWeb. I'm into music, video games and anime."
        ],
        "photo": [
          "https://toby3d.me/photo.jpg"
        ],
        "url": [
          "https://toby3d.me/"
        ]
      }
    }
  ],
  "rels": {
    ...
  },
  "rel-urls": {
    ...
  }
}

This error occurred on our Gitlab CI system with jobs running on Google Cloud.

go: dmitri.shuralyov.com/html/[email protected]: unrecognized import path "dmitri.shuralyov.com/html/belt" (https fetch: Get https://dmitri.shuralyov.com/html/belt?go-get=1: dial tcp 172.93.50.41:443: i/o timeout)
go: error loading module requirements

It's happened before in #25 and #23

It seems to be an intermittent failure and I can't reproduce it using curl from my own network.

curl -v 'https://dmitri.shuralyov.com/html/belt?go-get=1'

<meta name="go-import" content="dmitri.shuralyov.com/html/belt git https://dmitri.shuralyov.com/html/belt">
<meta name="go-import" content="dmitri.shuralyov.com/html/belt mod https://dmitri.shuralyov.com/api/module">
* Connection #0 to host dmitri.shuralyov.com left intact
<meta name="go-source" content="dmitri.shuralyov.com/html/belt https://dmitri.shuralyov.com/html/belt https://gotools.org/dmitri.shuralyov.com/html/belt https://gotools.org/dmitri.shuralyov.com/html/belt#{file}-L{line}">

Right now, the blog at https://dmitri.shuralyov.com/blog shows all blog posts one after another on one page. Some posts are small, some are very long. It makes it hard to skim the blog and tell where one post ends and another begins, since each post has a different and unpredictable size.

I should shorten long posts and add a "View Full Post" link at the bottom if it has been shortened. Perhaps with a fade to white effect to serve as a visual cue that the post has been cut off.

That way, a long blog post such as this one can be shown like this:

Currently, code discovery happens only once, during process startup:

No rediscovery is performed afterwards. As a result, any new Go packages that are created while the process is running do not appear until the process is restarted. Similarly, removed Go packages do not disappear until restart.

In order to improve the user experience, they should be discovered automatically. Rediscovery should happen when:

• a new git commit is pushed that contains new Go packages (or deletes some)
• a new repository is created

At this time, there is no general-purpose mechanism to move or delete repositories. Because they're very rare, such operations will continue to be done ad hoc manually and are considered out of scope for this issue for now.

Currently, it is possibly to sign in to home only via GitHub. While this has worked well and provides a simple user experience, it has some downsides:

• requires having a GitHub account
• doesn't work when github.com is down or unavailable

I want to make it so there isn't a single point of failure and so people can sign in without having a GitHub account, but I don't want to compromise on the user experience.

The following options are not great:

• usernames and passwords - no one wants to create and remember those (especially for a small site they visit rarely); susceptible to name squatting
• additional sign in providers - it becomes hard to remember which of multiple sign in providers you used last time, leading to a bad user experience

Instead, I plan to resolve this by switching to a URL-based sign in flow, and adding support for signing in via the IndieAuth protocol. IndieAuth is a decentralized identity protocol built on top of OAuth 2.0. Its latest specification is available at https://indieauth.spec.indieweb.org.

To sign in, a user will enter a URL they control. If they want to continue to sign in via GitHub, they can enter a URL like https://github.com/dmitshur:

When entering a github.com user profile URL, GitHub will be used to authenticate as before.

Users will also be able to sign in with any other URL they control that supports IndieAuth, such as https://example.com. This URL can be short and memorable, like one's personal website.

Similar to #23 and #25, build errors when trying to download packages from your site.

go: dmitri.shuralyov.com/app/[email protected]: unrecognized import path "dmitri.shuralyov.com/app/changes" (https fetch: Get https://dmitri.shuralyov.com/app/changes?go-get=1: dial tcp 172.93.50.41:443: i/o timeout)

`curl -v https://dmitri.shuralyov.com

• About to connect() to dmitri.shuralyov.com port 443
• Trying 172.93.50.41... Connection timed out
• couldn't connect to host
• Closing connection #0
  curl: (7) couldn't connect to host`