All sites I found under diputados.gob.mx are currently down Appears to be a nice s

Mexico: Deputies about crawler-planning HOT 2 CLOSED

opensanctions commented on September 23, 2024

Mexico: Deputies

from crawler-planning.

Comments (2)

bgmello commented on September 23, 2024

I wasn't able to open the website in my personal computer using safari, but for some reason it works using tor and when I tried to cURL from a AWS EC2. The below code get's the information already as a JSON:

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0',
    'Accept': 'application/json, text/plain, */*',
    'Accept-Language': 'en-US,en;q=0.5',
    # 'Accept-Encoding': 'gzip, deflate, br',
    'Referer': 'https://web.diputados.gob.mx/',
    'Content-Type': 'application/json',
    'Origin': 'https://web.diputados.gob.mx',
    'Connection': 'keep-alive',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-site',
}

json_data = {
    'operationName': None,
    'variables': {},
    'query': '{\n  allDiputados {\n    Oid\n    Nombre\n    PrimerApellido\n    SegundoApellido\n    NombreCompleto\n    Estado\n    Partido\n    Distrito\n    Legislacion\n    PrimerApellido\n    CabeceraMunicipal\n    Suplente\n    id_dip\n    IdDiputado\n    Correo\n    Telefono\n    TipoEleccion\n    Licencia\n    __typename\n  }\n}\n',
}

response = requests.post('https://micrositios.diputados.gob.mx:4001/graphql', headers=headers, json=json_data)

I can write the crawler and test it in a server, but I'm not sure it will run in the open sanction infrastructure, what do you think? @jbothma

This is the returned data: https://pastebin.com/7DhAcwWt

from crawler-planning.