imthaghost / goclone Goto Github PK

you say brew tap then download
what is brew tap can someone help me download it

I was experimenting with your 'goclone' and it seems to only clone the home page, not an entire site.
It doesn't seem to even attempt to crawl site links to clone more pages of the site.

Panic error when find data:image/png;base64, hash and try to download.

Aaaaaaaaaaaaaaaaaaaaa HELPHELPHELP

tried this today on zerohedge.com. works alright but the website is not a carbon copy.

This looks like it should clone any website, did i do something wrong?

No command goclone found, did you mean:
Command rclone in package rclone

Thanks for the wonderful work. However, when I try to clone a website protected by cloudflare anti-bot, this software failed to bypass these protection, hence not able to retrieve the useful real website.

Once downloaded website.. which files or scripts need to be removed? Specially, analytics content?

url: https://shtheme.com/demosd/foliox

then after its done , 95% of the files are not saved

css

js folder is empty

most of the images tend to get saved, but some are missing

I get "403 - Forbidden | Access to this page is forbidden." when trying to open the created URL. Any ideas?

The installable is not available on macports. Brew isn't compatible on most Macs.

When I clone the website with the amp website, the assets image is not downloaded using the amp tag

For those who don't use 'go', it's a tricky to google thing. I wasn't aware this was related to the go language and assumed it was some other utility program.

Then, the instructions list the out of date way to use the go get command first, and don't make it immediately clear that alternate instructions are coming afterwards.

I think i did all the steps:
1> Installed GO and added the PATH environment variable on $HOME/.profile
go version go1.21.6 linux/amd64

2> Did the manual installation of goclone
go install github.com/imthaghost/goclone/cmd/goclone@latest

3> Then tried the 'goclone <example.com>' and got the "command not found" message.

What am i missing?
Thanks!

My newly installed go version is go1.16.9

I can only download one web page by using the command. Can't the crawler traverse all the pages of the website

I realize this might be dangerous if someone haphazardly attempts WIkipedia or something, but it would be nice to have an -r flag to recursively follow all tags and clone subpages

after : goclone https://configtree.co
what path goclone saved the file?

panic: open C:\Users\difar\Downloads\goclone-master\goclone-master\cmd/www.censor.id/imgs/image?url=https%3A%2F%2Fwww.censor.id%2Fsites%2Fdefault%2Ffiles%2F2023-07%2Fcensor_img_230731212052.png: The filename, directory name, or volume label syntax is incorrect.

how do you pass multiple cookies?

Hey, so instead of showing web index.html shows me code. Maybe im stupid and dont understand something, but im new to coding could u help?

Edit

https://www.instagram.com

after installing Goclone I tried to copy a website but it's showing the error below. I am using WSL in windows 10. Please is there a way to resolve this or am I doing something wrong?

2021/08/03 10:22:39 mkdir /mnt/c/Windows/system32/tesla.com: permission denied
2021/08/03 10:22:39 mkdir /mnt/c/Windows/system32/tesla.com: permission denied
2021/08/03 10:22:39 mkdir /mnt/c/Windows/system32/tesla.com: permission denied
2021/08/03 10:22:39 mkdir /mnt/c/Windows/system32/tesla.com: permission denied
2021/08/03 10:22:39 open /mnt/c/Windows/system32/tesla.com/index.html: no such file or directory
Extracting --> https://tesla.com
panic: open /mnt/c/Windows/system32/tesla.com/index.html: no such file or directory

goroutine 18 [running]:
github.com/imthaghost/goclone/crawler.HTMLExtractor(0xc0002a42a0, 0x11, 0xc0000fa180, 0x21)
/Users/ghost/go/src/github.com/imthaghost/goclone/crawler/html.go:26 +0x2d3
github.com/imthaghost/goclone/crawler.Collector.func4(0xc000198780)
/Users/ghost/go/src/github.com/imthaghost/goclone/crawler/collector.go:52 +0xc4
github.com/gocolly/colly.(*Collector).handleOnRequest(0xc0000bb860, 0xc000198780)
/Users/ghost/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:924 +0x66
github.com/gocolly/colly.(*Collector).fetch(0xc0000bb860, 0xc0002a4260, 0x11, 0xa85a26, 0x3, 0x1, 0x0, 0x0, 0xc0001df7f0, 0xc0002b8000, ...)
/Users/ghost/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:586 +0x176
created by github.com/gocolly/colly.(*Collector).scrape
/Users/ghost/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:532 +0x3b1

How can I use goclone to extract content of internal-websites? I can put it in a struct like below, and then inject the json to elasic to for building site search functionality.

P.S: I'm new to golang. :)

/**
Plan : Import this struct in a crawling program, extract just text content no images/js, index for sitesearch.
**/
type WebPage struct {
	Id      string    `json:"id"`      // some id string or number
	Url     string    `json:"url"`     // URL of the page to index
	Title   string    `json:"title"`   // Title of the page
	Content string    `json:"content"` // Page content //TODO remove javascript, try to extract only core content
	Time    time.Time `json:"time"`    // TODO timestamp of page creation time
}

func (document *WebPage) Print() {
	// enc := json.NewEncoder(os.Stdout)
	// enc.SetIndent("", "  ")
	// document.Content = ""
	// enc.Encode(document)
	println(fmt.Sprintf("page:  {\n  title: %s, \n  url : %s, \n  content:%s, \n  time:%s \n}", document.Title, document.Url, "-redacted-", document.Time))
}

As going step by step through installation manual I got stuck over this issue of which I attached screenshot.

C:\Users\hp\go/src/github.com/imthaghost/goclone/cmd/goclone

Following the project readme's manual install instructions failed.

Go Version: 1.18.4

Command ran + output:

go get github.com/imthaghost/goclone

go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

Alternative that also failed:

go install github.com/imthaghost/goclone@latest

Usage:
goclone https://www.instagram.com/ [flags]

Flags:
-C, --cookie strings Pre-set these cookies
-h, --help help for goclone
-o, --open Automatically open project in deafult browser
-p, --proxy_string string Proxy connection string. Support http and socks5 https://pkg.go.dev/github.com/gocolly/colly#Collector.SetProxy
-s, --serve Serve the generated files using Echo.
-u, --user_agent string Custom User Agent

Originally posted by @Hugorino11 in #36 (comment)

I also want to clone files with other extensions. I'm trying to help someone troubleshoot a website they're building on another platform, and it's not easy to work on their stuff and would be easier to work with raw files for me. Their server doesn't have permissive CORS headers, though, and the site owner has no idea how to change that. They specifically have several files with .glb and .gltf extensions.

Would be nice to be able to specify extensions to include.

Hey there, thanks for this, looks great so far!

Got this error while downloading a site...

Img found --> blob:https://twitter.com/c86b1e90-025d-4bfa-9844-adb20bc84bf1
Extracting -->  blob:https://twitter.com/c86b1e90-025d-4bfa-9844-adb20bc84bf1
panic: Get "blob:https://twitter.com/c86b1e90-025d-4bfa-9844-adb20bc84bf1": unsupported protocol scheme "blob"

and it died. Feature request, support for "blob" and "data"

unsupported protocol scheme "data"

I added in a string replace for now but I'm not good enough at go to be worthy of a PR yet. :-)

Thanks!

it does not work, it can clone some websites but not all.

how do i uninstall all the installed packages or where are they downloaded to just in the main folder somewhere?