coxier / cospider Goto Github PK

CoSpider是一个专注**大学课表的爬虫，类似于超级课程表,目前能爬取华中科技大学、武汉理工大学的个人课表。你可以用它来做校园通、课程表、验证学生身份等等

• 网络工具：Firefox + Firebug
• 开发工具：IDEA
• 构建工具： gradle

• OkHttp3.0
• Jsoup

想借鉴此库来爬取你们学校的课表吗？ 一般来说爬课表，关键的是模拟登陆，让服务器认为你是浏览器，使用OkHttp3.0管理Cookie即可，这一点请参考 CustomCookieJar 开发步骤：

• 实现LoginTask接口，完成登录步骤
• 实现CouseTask接口，完成爬取课表步骤

//new a login task implemnts the interface
class YourLoginTask implements LoginTask{
     int execute(OkHttpClient client, String userName, String password){
        // just try to login
     }
}


// couser task
class YourCourseTask implements CourseTask{
     String execute(OkHttpClient client, String start, String end){
        // get course and return it
     }
}

使用方法(以爬取华科课表为例)：

Spider spider = new Spider.Builder()
                .addName("username")
                .addPassword("password")
                .addStart("2016-10-19")
                .addEnd("2016-10-20")
                .build();
HubLoginTask loginTask = new HubLoginTask();
HubCourseTask courseTask = new HubCourseTask();
Task task = new Task(loginTask, courseTask);
String result = spider.execute(task);

• 华中科技大学HUB系统：密码采用Base64加密，而且使用了中转登录机制
• 对于武大和华农之类需要验证码的：在手机端或者Pc端存储验证码图片，然后手动填写验证码

Copyright 2016 coxier.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.