jimmylaurent / cloudflare-scraper Goto Github PK

View Code? Open in Web Editor NEW

281.0 281.0 29.0 83 KB

A package to bypass Cloudflare's protection

License: MIT License

JavaScript 100.00%

cloudflare-scraper's People

Stargazers

Watchers

cloudflare-scraper's Issues

Can we use it with Playwright functions?

Creating types for TypeScript

Hello. The thing is, I want to use your this module in TypeScript, but I can't do it because there are no types. I would be very grateful if they were created ❤

It's possible to edit User Agent ?

Hi, I tried and everything is ok bypassing a cloudflare frlm a web from a local server, but when I used It from heroku server, I got and error with the response with the html of the cloudfare web. There could be something to modify to get compatibility with heroku git?

I can not send POST request

I have problem with sending post request, i tried already everything and i have no idea how to do it.

I tried this and some other combinations, but doesnt work...

const response = await cloudflareScraper.post('https://www.xxxxxxxxxxx',  { 'postData': 'name=Adam&pass=123123' });

your contact

Mind Sharing your telegram or Discord???

CF updated security

After CF updated I getting this errror : "timeout on just a moment" . I think need update

Doesn't work: Still having Cloudflare protection screen with title "Just a moment..."

The lib doesn't work, I still having Cloudflare protection screen with title "Just a moment..."
I took a look at the resulting HTML from cromium and there is content there - it works
But got gets 403 result with that waiting page

Any ideas of how to fix?

About the solution to fix this package through js challenge

Recently, I am also doing related research and found that this package only needs to modify the conditions of the detection part of the js challenge slightly to pass the js challenge normally:
In src/utils.js

  function isCloudflareJSChallenge(body) {
-   return body.includes('managed_checking_msg');
+   return body.includes('/cdn-cgi/images/trace/jsch');
  }

Cloudflare scraper can not work on Centos 7

I got connection refused when run bypass cloudflare on Centos, Could you help me check this? Please following image as below:

403 captcha bypass

Customization features

Hi, thanks for creating such a wonderful module

I am wondering if it's possible to add the following features:

Setting your own user-agent
Setting your own proxy
Receiving the cookies in a json format instead of full HTML. That's what also #8 meant I think

I'll also be looking if I can contribute to this repo

how to add a proxy ?

how to add proxy on request?

How does it exactly work?

I dont get it. So cloudflare doesnt need to validate your browser? You are just copying/altering expires cookies when accessing cloudfront page. So there is a cookie that says if cloudflare shoudl be invoked?
What if we enter cloudflare for the first time before ever accessing the webpage?

str.trim is not a function

Hi @JimmyLaurent and thanks for sharing,

I am trying const response = await cloudflareScraper.get('https://www.malt.fr'); in order to scrape my profile, and this is the error: str.trim is not a function

It seems this happen in [email protected]

Can this be fixed ? Because it's actually the only client I have found that actually have bypassed the cloudflare security but this error makes it unusuable.

Thanks for reading.

Support remote Chrome

Hi,
is it possible to add support for remote Chrome? (ex: running in Docker)

I'm running cloudflare-scraper on headless machine (no X, no Chrome installed) and I'd like to solve Cloudflare puzzle, retrieve cookie and continue my scrapping using different tool.
I was able to do it, but I needed to do some dirty code patching first.

Error: timeout on just a moment

any ideas?

How to run

Hi,

im not sure how I can get the bot setup and ran. Please help

What exactly does this do?

Can you please add some more details to the readme.

Puppeteer (chromium headless) is used to retrieve cloudflare cookies then request module is used to perform requests making this solution reliable but also pretty fast.

What are some use-cases? Is this a workaround for cloudflare detection?

fixed

Hello sir, how can use your script to bypass new uam challenge?
here is my exemple code, using cloudscraper
// fixed

Problem: library won't work

It seems this library won't work.

Testing on https://www.ckgsir.com/bluestrip-widget

const cloudflareScraper = require('cloudflare-scraper');

(async () => {
    try {
        const response = await cloudflareScraper.get('https://www.ckgsir.com/bluestrip-widget');
        console.log(response);
    } catch (error) {
        console.log(error);
    }
})();

C:\Users\Max\JS>node test.js
StatusCodeError: 403 - "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]>    <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n  if (!navigator.cookieEnabled) {\n    window.addEventListener('DOMContentLoaded', function () {\n      var cookieEl = document.getElementById('cookie-alert');\n      cookieEl.style.display = 'block';\n    })\n  }\n</script>\n<!--<![endif]-->\n\n\n</head>\n<body>\n  <div id=\"cf-wrapper\">\n    <div class=\"cf-alert cf-alert-error cf-cookie-error\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>\n    <div id=\"cf-error-details\" class=\"cf-error-details-wrapper\">\n      <div class=\"cf-wrapper cf-header cf-error-overview\">\n        <h1 data-translate=\"block_headline\">Sorry, you have been blocked</h1>\n        <h2 class=\"cf-subheadline\"><span data-translate=\"unable_to_access\">You are unable to access</span> ckgsir.com</h2>\n      </div><!-- /.header -->\n\n      <div class=\"cf-section cf-highlight\">\n        <div class=\"cf-wrapper\">\n          <div class=\"cf-screenshot-container cf-screenshot-full\">\n            \n              <span class=\"cf-no-screenshot error\"></span>\n            \n          </div>\n        </div>\n      </div><!-- /.captcha-container -->\n\n      <div class=\"cf-section cf-wrapper\">\n        <div class=\"cf-columns two\">\n          <div class=\"cf-column\">\n            <h2 data-translate=\"blocked_why_headline\">Why have I been blocked?</h2>\n\n            <p data-translate=\"blocked_why_detail\">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>\n          </div>\n\n          <div class=\"cf-column\">\n            <h2 data-translate=\"blocked_resolve_headline\">What can I do to resolve this?</h2>\n\n            <p data-translate=\"blocked_resolve_detail\">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>\n          </div>\n        </div>\n      </div><!-- /.section -->\n\n      <div class=\"cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300\">\n  <p class=\"text-13\">\n    <span class=\"cf-footer-item sm:block sm:mb-1\">Cloudflare Ray ID: <strong class=\"font-semibold\">760d28b3ab18903d</strong></span>\n    <span class=\"cf-footer-separator sm:hidden\">&bull;</span>\n    <span id=\"cf-footer-item-ip\" class=\"cf-footer-item hidden sm:block sm:mb-1\">\n      Your IP:\n      <button type=\"button\" id=\"cf-footer-ip-reveal\" class=\"cf-footer-ip-reveal-btn\">Click to reveal</button>\n      <span class=\"hidden\" id=\"cf-footer-ip\">86.57.68.238</span>\n      <span class=\"cf-footer-separator sm:hidden\">&bull;</span>\n    </span>\n    <span class=\"cf-footer-item sm:block sm:mb-1\"><span>Performance &amp; security by</span> <a rel=\"noopener noreferrer\" href=\"https://www.cloudflare.com/5xx-error-landing\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>\n    \n  </p>\n  <script>(function(){function d(){var b=a.getElementById(\"cf-footer-item-ip\"),c=a.getElementById(\"cf-footer-ip-reveal\");b&&\"classList\"in b&&(b.classList.remove(\"hidden\"),c.addEventListener(\"click\",function(){c.classList.add(\"hidden\");a.getElementById(\"cf-footer-ip\").classList.remove(\"hidden\")}))}var a=document;document.addEventListener&&a.addEventListener(\"DOMContentLoaded\",d)})();</script>\n</div><!-- /.error-footer -->\n\n\n    </div><!-- /#cf-error-details -->\n  </div><!-- /#cf-wrapper -->\n\n  <script>\n  window._cf_translation = {};\n  \n  \n</script>\n\n</body>\n</html>\n"
    at new StatusCodeError (C:\Users\Max\JS\node_modules\request-promise-core\lib\errors.js:32:15)
    at Request.plumbing.callback (C:\Users\Max\JS\node_modules\request-promise-core\lib\plumbing.js:104:33)
    at Request.RP$callback [as _callback] (C:\Users\Max\JS\node_modules\request-promise-core\lib\plumbing.js:46:31)
    at Request.self.callback (C:\Users\Max\JS\node_modules\request\request.js:185:22)
    at Request.emit (node:events:527:28)
    at Request.<anonymous> (C:\Users\Max\JS\node_modules\request\request.js:1154:10)
    at Request.emit (node:events:527:28)
    at Gunzip.<anonymous> (C:\Users\Max\JS\node_modules\request\request.js:1076:12)
    at Object.onceWrapper (node:events:641:28)
    at Gunzip.emit (node:events:527:28) {
  statusCode: 403,
  error: '<!DOCTYPE html>\n' +
    '<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->\n' +
    '<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->\n' +
    '<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->\n' +
    '<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->\n' +
    '<head>\n' +
    '<title>Attention Required! | Cloudflare</title>\n' +
    '<meta charset="UTF-8" />\n' +
    '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\n' +
    '<meta http-equiv="X-UA-Compatible" content="IE=Edge" />\n' +
    '<meta name="robots" content="noindex, nofollow" />\n' +
    '<meta name="viewport" content="width=device-width,initial-scale=1" />\n' +
    '<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" />\n' +
    `<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]-->\n` +
    '<style>body{margin:0;padding:0}</style>\n' +
    '\n' +
    '\n' +
    '<!--[if gte IE 10]><!-->\n' +
    '<script>\n' +
    '  if (!navigator.cookieEnabled) {\n' +
    "    window.addEventListener('DOMContentLoaded', function () {\n" +
    "      var cookieEl = document.getElementById('cookie-alert');\n" +
    "      cookieEl.style.display = 'block';\n" +
    '    })\n' +
    '  }\n' +
    '</script>\n' +
    '<!--<![endif]-->\n' +
    '\n' +
    '\n' +
    '</head>\n' +
    '<body>\n' +
    '  <div id="cf-wrapper">\n' +
    '    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>\n' +
    '    <div id="cf-error-details" class="cf-error-details-wrapper">\n' +
    '      <div class="cf-wrapper cf-header cf-error-overview">\n' +
    '        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>\n' +
    '        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> ckgsir.com</h2>\n' +
    '      </div><!-- /.header -->\n' +
    '\n' +
    '      <div class="cf-section cf-highlight">\n' +
    '        <div class="cf-wrapper">\n' +
    '          <div class="cf-screenshot-container cf-screenshot-full">\n' +
    '            \n' +
    '              <span class="cf-no-screenshot error"></span>\n' +
    '            \n' +
    '          </div>\n' +
    '        </div>\n' +
    '      </div><!-- /.captcha-container -->\n' +
    '\n' +
    '      <div class="cf-section cf-wrapper">\n' +
    '        <div class="cf-columns two">\n' +
    '          <div class="cf-column">\n' +
    '            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>\n' +
    '\n' +
    '            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>\n' +
    '          </div>\n' +
    '\n' +
    '          <div class="cf-column">\n' +
    '            <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>\n' +
    '\n' +
    '            <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>\n' +
    '          </div>\n' +
    '        </div>\n' +
    '      </div><!-- /.section -->\n' +
    '\n' +
    '      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">\n' +
    '  <p class="text-13">\n' +
    '    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">760d28b3ab18903d</strong></span>\n' +
    '    <span class="cf-footer-separator sm:hidden">&bull;</span>\n' +
    '    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">\n' +
    '      Your IP:\n' +
    '      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>\n' +
    '      <span class="hidden" id="cf-footer-ip">86.57.68.238</span>\n' +
    '      <span class="cf-footer-separator sm:hidden">&bull;</span>\n' +
    '    </span>\n' +
    '    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>\n' +
    '    \n' +
    '  </p>\n' +
    '  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>\n' +
    '</div><!-- /.error-footer -->\n' +
    '\n' +
    '\n' +
    '    </div><!-- /#cf-error-details -->\n' +
    '  </div><!-- /#cf-wrapper -->\n' +
    '\n' +
    '  <script>\n' +
    '  window._cf_translation = {};\n' +
    '  \n' +
    '  \n' +
    '</script>\n' +
    '\n' +
    '</body>\n' +
    '</html>\n',
  options: {
    jar: RequestJar { _jar: [CookieJar] },
    headers: {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'
    },
    gzip: true,
    uri: 'https://www.ckgsir.com/bluestrip-widget',
    method: 'GET',
    callback: [Function: RP$callback],
    transform: undefined,
    simple: true,
    resolveWithFullResponse: false,
    transform2xxOnly: false
  },
  response: <ref *1> IncomingMessage {
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 16384,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: [],
      flowing: false,
      ended: true,
      endEmitted: true,
      reading: false,
      constructed: true,
      sync: false,
      needReadable: false,
      emittedReadable: false,
      readableListening: false,
      resumeScheduled: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      destroyed: true,
      errored: null,
      closed: true,
      closeEmitted: true,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: false,
      dataEmitted: true,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: true
    },
    _events: [Object: null prototype] {
      end: [Array],
      close: [Function (anonymous)]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'www.ckgsir.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'www.ckgsir.com',
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: null,
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [ClientRequest],
      [Symbol(res)]: null,
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 8,
      [Symbol(kHandle)]: null,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: false,
      [Symbol(kSetKeepAliveInitialDelay)]: 0,
      [Symbol(kBytesRead)]: 2452,
      [Symbol(kBytesWritten)]: 235,
      [Symbol(connect-options)]: [Object],
      [Symbol(RequestTimeout)]: undefined
    },
    httpVersionMajor: 1,
    httpVersionMinor: 1,
    httpVersion: '1.1',
    complete: true,
    rawHeaders: [
      'Date',
      'Thu, 27 Oct 2022 17:30:20 GMT',
      'Content-Type',
      'text/html; charset=UTF-8',
      'Transfer-Encoding',
      'chunked',
      'Connection',
      'close',
      'Cache-Control',
      'max-age=15',
      'Expires',
      'Thu, 27 Oct 2022 17:30:35 GMT',
      'X-Frame-Options',
      'SAMEORIGIN',
      'Set-Cookie',
      '__cf_bm=lrsj4zmz_62ze_mAsNS9cKeZrrEY3T8JkGUnB8hFX6g-1666891820-0-AX5PDllPO/nNYQDHTpxXeXHPGtguDVxFdWBB7jveBgj8PbB9DgJis7vLv935h1v8Wl4gr6CnSFKrh5HmH6IedyA=; path=/; expires=Thu, 27-Oct-22 18:00:20 GMT; domain=.ckgsir.com; HttpOnly; Secure; SameSite=None',
      'Vary',
      'Accept-Encoding',
      'Strict-Transport-Security',
      'max-age=0; includeSubDomains; preload',
      'X-Content-Type-Options',
      'nosniff',
      'Server',
      'cloudflare',
      'CF-RAY',
      '760d28b3ab18903d-FRA',
      'Content-Encoding',
      'gzip'
    ],
    rawTrailers: [],
    aborted: false,
    upgrade: false,
    url: '',
    method: null,
    statusCode: 403,
    statusMessage: 'Forbidden',
    client: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'www.ckgsir.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'www.ckgsir.com',
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: null,
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [ClientRequest],
      [Symbol(res)]: null,
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 8,
      [Symbol(kHandle)]: null,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: false,
      [Symbol(kSetKeepAliveInitialDelay)]: 0,
      [Symbol(kBytesRead)]: 2452,
      [Symbol(kBytesWritten)]: 235,
      [Symbol(connect-options)]: [Object],
      [Symbol(RequestTimeout)]: undefined
    },
    _consuming: true,
    _dumped: false,
    req: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'GET /bluestrip-widget HTTP/1.1\r\n' +
        'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36\r\n' +
        'host: www.ckgsir.com\r\n' +
        'accept-encoding: gzip, deflate\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/bluestrip-widget',
      _ended: true,
      res: [Circular *1],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'www.ckgsir.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    request: Request {
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      headers: [Object],
      gzip: true,
      uri: [Url],
      method: 'GET',
      readable: true,
      writable: true,
      explicitMethod: true,
      _qs: [Querystring],
      _auth: [Auth],
      _oauth: [OAuth],
      _multipart: [Multipart],
      _redirect: [Redirect],
      _tunnel: [Tunnel],
      _rp_resolve: [Function (anonymous)],
      _rp_reject: [Function (anonymous)],
      _rp_promise: [Promise],
      _rp_callbackOrig: undefined,
      callback: [Function (anonymous)],
      _rp_options: [Object],
      setHeader: [Function (anonymous)],
      hasHeader: [Function (anonymous)],
      getHeader: [Function (anonymous)],
      removeHeader: [Function (anonymous)],
      localAddress: undefined,
      pool: {},
      dests: [],
      __isRequestRequest: true,
      _callback: [Function: RP$callback],
      proxy: null,
      tunnel: true,
      setHost: true,
      originalCookieHeader: undefined,
      _jar: [RequestJar],
      port: 443,
      host: 'www.ckgsir.com',
      path: '/bluestrip-widget',
      httpModule: [Object],
      agentClass: [Function: Agent],
      agent: [Agent],
      _started: true,
      href: 'https://www.ckgsir.com/bluestrip-widget',
      req: [ClientRequest],
      ntick: true,
      response: [Circular *1],
      originalHost: 'www.ckgsir.com',
      originalHostHeaderName: 'host',
      responseContent: [Gunzip],
      _ended: true,
      _destdata: true,
      _callbackCalled: true,
      [Symbol(kCapture)]: false
    },
    toJSON: [Function: responseToJSON],
    caseless: Caseless { dict: [Object] },
    body: '<!DOCTYPE html>\n' +
      '<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->\n' +
      '<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->\n' +
      '<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->\n' +
      '<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->\n' +
      '<head>\n' +
      '<title>Attention Required! | Cloudflare</title>\n' +
      '<meta charset="UTF-8" />\n' +
      '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\n' +
      '<meta http-equiv="X-UA-Compatible" content="IE=Edge" />\n' +
      '<meta name="robots" content="noindex, nofollow" />\n' +
      '<meta name="viewport" content="width=device-width,initial-scale=1" />\n' +
      '<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" />\n' +
      `<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]-->\n` +
      '<style>body{margin:0;padding:0}</style>\n' +
      '\n' +
      '\n' +
      '<!--[if gte IE 10]><!-->\n' +
      '<script>\n' +
      '  if (!navigator.cookieEnabled) {\n' +
      "    window.addEventListener('DOMContentLoaded', function () {\n" +
      "      var cookieEl = document.getElementById('cookie-alert');\n" +
      "      cookieEl.style.display = 'block';\n" +
      '    })\n' +
      '  }\n' +
      '</script>\n' +
      '<!--<![endif]-->\n' +
      '\n' +
      '\n' +
      '</head>\n' +
      '<body>\n' +
      '  <div id="cf-wrapper">\n' +
      '    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>\n' +
      '    <div id="cf-error-details" class="cf-error-details-wrapper">\n' +
      '      <div class="cf-wrapper cf-header cf-error-overview">\n' +
      '        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>\n' +
      '        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> ckgsir.com</h2>\n' +
      '      </div><!-- /.header -->\n' +
      '\n' +
      '      <div class="cf-section cf-highlight">\n' +
      '        <div class="cf-wrapper">\n' +
      '          <div class="cf-screenshot-container cf-screenshot-full">\n' +
      '            \n' +
      '              <span class="cf-no-screenshot error"></span>\n' +
      '            \n' +
      '          </div>\n' +
      '        </div>\n' +
      '      </div><!-- /.captcha-container -->\n' +
      '\n' +
      '      <div class="cf-section cf-wrapper">\n' +
      '        <div class="cf-columns two">\n' +
      '          <div class="cf-column">\n' +
      '            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>\n' +
      '\n' +
      '            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>\n' +
      '          </div>\n' +
      '\n' +
      '          <div class="cf-column">\n' +
      '            <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>\n' +
      '\n' +
      '            <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>\n' +
      '          </div>\n' +
      '        </div>\n' +
      '      </div><!-- /.section -->\n' +
      '\n' +
      '      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">\n' +
      '  <p class="text-13">\n' +
      '    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">760d28b3ab18903d</strong></span>\n' +
      '    <span class="cf-footer-separator sm:hidden">&bull;</span>\n' +
      '    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">\n' +
      '      Your IP:\n' +
      '      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>\n' +
      '      <span class="hidden" id="cf-footer-ip">86.57.68.238</span>\n' +
      '      <span class="cf-footer-separator sm:hidden">&bull;</span>\n' +
      '    </span>\n' +
      '    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>\n' +
      '    \n' +
      '  </p>\n' +
      '  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>\n' +
      '</div><!-- /.error-footer -->\n' +
      '\n' +
      '\n' +
      '    </div><!-- /#cf-error-details -->\n' +
      '  </div><!-- /#cf-wrapper -->\n' +
      '\n' +
      '  <script>\n' +
      '  window._cf_translation = {};\n' +
      '  \n' +
      '  \n' +
      '</script>\n' +
      '\n' +
      '</body>\n' +
      '</html>\n',
    [Symbol(kCapture)]: false,
    [Symbol(kHeaders)]: {
      date: 'Thu, 27 Oct 2022 17:30:20 GMT',
      'content-type': 'text/html; charset=UTF-8',
      'transfer-encoding': 'chunked',
      connection: 'close',
      'cache-control': 'max-age=15',
      expires: 'Thu, 27 Oct 2022 17:30:35 GMT',
      'x-frame-options': 'SAMEORIGIN',
      'set-cookie': [Array],
      vary: 'Accept-Encoding',
      'strict-transport-security': 'max-age=0; includeSubDomains; preload',
      'x-content-type-options': 'nosniff',
      server: 'cloudflare',
      'cf-ray': '760d28b3ab18903d-FRA',
      'content-encoding': 'gzip'
    },
    [Symbol(kHeadersCount)]: 28,
    [Symbol(kTrailers)]: null,
    [Symbol(kTrailersCount)]: 0,
    [Symbol(RequestTimeout)]: undefined
  }
}

Error: could't find the site key

Hi, I have been using cloudflare-scraper for around three-four months, with a forked repository to change the linux agent user, but the last week has started to failing with the next error:

2021-05-10T17:44:49.403793+00:00 app[worker.1]: Error: could't find the site key
2021-05-10T17:44:49.403805+00:00 app[worker.1]: at extract (/app/node_modules/cloudflare-scraper/src/utils.js:25:11)
2021-05-10T17:44:49.403806+00:00 app[worker.1]: at handleCaptcha (/app/node_modules/cloudflare-scraper/src/handleCaptcha.js:39:19)
2021-05-10T17:44:49.403806+00:00 app[worker.1]: at fillCookiesJar (/app/node_modules/cloudflare-scraper/src/fillCookiesJar.js:52:15)

As you can see I am using It through heroku service, and eventually works fine since my computer but no through heroku, even after restart the application.

It would be the problem that cloudflare ban de ip's from heroku servers?, there is some moments than It works from my computer and anothers times that not.

I really appreciate your response.

Does it work with puppeteer?

how to add proxy?

how to add proxy to the request?

Cookies

How can i get cookies after successful bypass?

ReferenceError: commonArgs is not defined

const response = await cloudflareScraper.get({
            url: url,
            proxy: 'http://127.0.0.1:8888',
            headers: {
                'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'
            }
        });

ReferenceError: commonArgs is not defined
    at createBrowser (/root/js/node_modules/cloudflare-scraper/src/createBrowser.js:35:5)
    at fillCookiesJar (/root/js/node_modules/cloudflare-scraper/src/fillCookiesJar.js:28:25)
    at handleError (/root/js/node_modules/cloudflare-scraper/index.js:18:11)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async cloudflareScraper (/root/js/node_modules/cloudflare-scraper/index.js:37:20)
    at async main (/root/js/dev2.js:9:26)

Can not bypass cloudflare captcha

I got 403 status code from CF when running the following code:

const cloudflareScraper = require('cloudflare-scraper');

(async () => {
  try {
    const response = await cloudflareScraper.get('https://widtodo.com/4c8rx0tt8ek4');
    console.log(response);
  } catch (error) {
    console.log(error)
  }
})();

packages.json

{
  "dependencies": {
    "cloudflare-scraper": "^1.0.4",
    "puppeteer": "^3.3.0"
  }
}

TypeError: browser.setMaxListeners is not a function

TypeError: browser.setMaxListeners is not a function
    at StealthPlugin.onBrowser (/root/js/node_modules/puppeteer-extra-plugin-stealth/index.js:155:13)
    at StealthPlugin._bindBrowserEvents (/root/js/node_modules/puppeteer-extra-plugin/dist/index.cjs.js:479:24)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async PuppeteerExtra.callPlugins (/root/js/node_modules/puppeteer-extra/dist/index.cjs.js:376:13)
    at async PuppeteerExtra.launch (/root/js/node_modules/puppeteer-extra/dist/index.cjs.js:131:9)
    at async createBrowser (/root/js/node_modules/cloudflare-scraper/src/createBrowser.js:103:10)
    at async fillCookiesJar (/root/js/node_modules/cloudflare-scraper/src/fillCookiesJar.js:28:19)
    at async handleError (/root/js/node_modules/cloudflare-scraper/index.js:18:5)
    at async cloudflareScraper (/root/js/node_modules/cloudflare-scraper/index.js:37:20)
    at async /root/js/dev_c.js:5:22

const cloudflareScraper = require('cloudflare-scraper');

(async () => {
  try {
    const response = await cloudflareScraper.get(url);
    console.log(response);
  } catch (error) {
    console.log(error);
  }
})();

Error: npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances

C:\Users\Max\JS>npm i cloudflare-scraper
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142

added 53 packages, and audited 158 packages in 12s

10 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

StatusCodeError: 403 - "error code: 1020"

StatusCodeError: 403 - "error code: 1020" -- error cloudfire

tunneling error when try with a proxy

error: Error: tunneling socket could not be established, cause=read ECONNRESET
      at ClientRequest.onError (C:\*****\node_modules\tunnel-agent\index.js:177:17)
      at Object.onceWrapper (events.js:422:26)
      at ClientRequest.emit (events.js:315:20)
      at Socket.socketErrorListener (_http_client.js:426:9)
      at Socket.emit (events.js:315:20)
      at emitErrorNT (internal/streams/destroy.js:92:8)
      at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
      at processTicksAndRejections (internal/process/task_queues.js:84:21) {
    code: 'ECONNRESET'
  },

code used

(async () => {
    try {
        const jar = cloudflareScraper.jar();
        const url = 'site';
        const response = await cloudflareScraper.get(url, {
            jar,
            headers: defaultHeaders,
            proxy: proxyServer
        });
        const cookiesObject = jar.getCookies(url);
        const cookieJson = JSON.stringify(cookieObject, null, 2);
        const cookieStr = jar.getCookieString(url);
        console.log(cookieStr);
    } catch (error) {
        console.log(`Error :`, error/*.statusCode, error.error*/);
    }
})()

Reduce cpu usage

It possible to reduce cpu usage? When i launch script in some threads on different sites - it usage 8/16 core/threads near on 80%

Can't retrieve HTML from some sites

Hi, I'm trying to get HTML from these links:

Unfortunately these sites have different Cloudflare Protection and no program can get HTML from there.

NB:
These sites can only be accessed via Italian IP, but attached are the errors recovered (with HTML no bypassed).
ilgeniodellostreaming.txt
igds.txt

Thank you in advance and have a nice day : )

i can't import lib with typescript

Can' open link

http://www.k51r.com/

Post metods whith data parametrs

const browser = await puppeteer.launch(LAUNCH_PUPPETEER_OPTS_ALL)
const pageSearch = await browser.newPage()
await pageSearch.setRequestInterception(true)
await pageSearch.on('request', interceptedRequest => {
let data = {
'method': 'POST',
'postData': payload,
'headers': {
...interceptedRequest.headers(),
"Content-Type": "application/json"
}
};
interceptedRequest.continue(data);
})
await pageSearch.goto(url)
const resultsCars = await pageSearch.content()
await browser.close()
...................................................
How to implement this with await cloudflareScraper.post (url, data)

why looking for "Just a moment..." in title instead of checking if the `cf-browser-verification` tag in the HTML code is present?

Cloudflare allows to change some part of the template of the verification page.
I think looking for the words "Just a moment..." in the title is not a good idea because a website could bypass your program just by changing the title of the verification page: https://github.com/JimmyLaurent/cloudflare-scraper/blob/master/src/getCookies.js#L24
It's better to look for cf-browser-verification or another "generic" tag in the HTML code because that is something that the owner can't change because it's part of the "verification" JavaScript code of Cloudflare:

let pageContent = await page.content();
while (pageContent.includes("cf-browser-verification")) {
    response = await page.waitForNavigation({
      timeout: 45000,
      waitUntil: 'domcontentloaded'
    });
    pageContent = await page.content();
    count++;
    if (count === 10) {
      throw new Error('timeout on just a moment');
    }
  }

Example click iframe captcha human

Your code is working Thanks !! Is there a solution for clicking validation iframe checkbox human

Cannot read property 'reduce' of undefined

TypeError: Cannot read property 'reduce' of undefined
    at tryToSolve (/root/dev/node_modules/hcaptcha-solver/index.js:79:25)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async solveCaptcha (/root/dev/node_modules/hcaptcha-solver/index.js:112:22)
    at async handleCaptcha (/root/dev/node_modules/cloudflare-scraper/src/handleCaptcha.js:48:7)
    at async fillCookiesJar (/root/dev/node_modules/cloudflare-scraper/src/fillCookiesJar.js:39:7)
    at async handleError (/root/dev/node_modules/cloudflare-scraper/index.js:18:5)
    at async cloudflareScraper (/root/dev/node_modules/cloudflare-scraper/index.js:37:20)
    at async /root/dev/1.js:5:22

const cloudflareScraper = require('cloudflare-scraper');
(async () => {
  try {
    const response = await cloudflareScraper.get(url, {
    });
    console.log(response);
  } catch (error) {
    console.log(error);
  }
})();

Cannot destructure property 'body' of 'error.response' as it is undefined

Sometimes catch this error when use proxy

TypeError: Cannot destructure property 'body' of 'error.response' as it is undefined.
    at isCloudflareIUAMError (/root/js/node_modules/cloudflare-scraper/index.js:8:11)
    at handleError (/root/js/node_modules/cloudflare-scraper/index.js:13:7)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async cloudflareScraper (/root/js/node_modules/cloudflare-scraper/index.js:34:20)
    at async /root/js/parse/test.js:10:32

how to get cookies and put it in default request?

I got the following situation: I receive cookies using your library, and then I need to send cookies in a regular request, for example, via request.get(), but when I do this these cookies do not pass. maybe i need to do something with jar? even though I tried...

How to use with aws ?

Help with some info

/Fixed

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.