自用Java爬虫工具JAVA-CURL已开源

摘要：工具和资料群高级爬虫作者自建群，欢迎加入作者收集的爬虫相关工具和资料简介类是以命令行工具为参考，使用标准的实现的工具类。

QQ群 - Javascript高级爬虫 - 作者自建群，欢迎加入！

awesome-java-crawler - 作者收集的爬虫相关工具和资料

CUrl类是以命令行工具CUrl为参考，使用标准Java的HttpURLConnection实现的Http工具类。

项目地址: https://github.com/rockswang/...
中央仓库: https://mvnrepository.com/art...

基于标准Java运行库的Http类实现，源码兼容级别为1.6，适用性广泛，可用于服务端、Android等Java环境

代码精简紧凑，仅一个1000余行的Java源文件，无任何外部依赖，可不用Maven直接源码级重用

完全兼容CUrl命令行工具的常用开关，可直接作为命令行工具替代之

支持所有HTTP Method，支持多文件上传

通过ThreadLocal解决了标准Java中Cookie只能全局保存的问题，可每线程独立维护Cookie

可将线程中保存的Cookies序列化保存，方便建立Cookies池

支持HTTP认证，支持HTTPS，可启用/忽略证书安全

支持每连接代理，支持需认证的HTTP/HTTPS代理

跳转行为可控制，可获取到每步跳转的应答头信息

支持编程自定义应答解析器

支持失败重试，可编程自定义可重试异常

例1：POST表单提交

    public void httpPost() {
        CUrl curl = new CUrl("http://httpbin.org/post")
                .data("hello=world&foo=bar")
                .data("foo=overwrite");
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例2：通过Fiddler代理（抓包工具）访问HTTPS站点

    public void insecureHttpsViaFiddler() {
        CUrl curl = new CUrl("https://httpbin.org/get")
                .proxy("127.0.0.1", 8888) // Use Fiddler to capture & parse HTTPS traffic
                .insecure();  // Ignore certificate check since it"s issued by Fiddler
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例3：上传多个文件，一个内存文件，一个物理文件

    public void uploadMultipleFiles() {
        CUrl.MemIO inMemFile = new CUrl.MemIO();
        try { inMemFile.getOutputStream().write("text file content blabla...".getBytes()); } catch (Exception ignored) {}
        CUrl curl = new CUrl("http://httpbin.org/post")
                .form("formItem", "value") // a plain form item
                .form("file", inMemFile)           // in-memory "file"
                .form("image", new CUrl.FileIO("D:	mpa2.png")); // A file in storage
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例4：模拟手机浏览器上的AJAX请求，添加自定义请求头

    public void customUserAgentAndHeaders() {
        String mobileUserAgent = "Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; KNT-AL10 Build/HUAWEIKNT-AL10) " 
                + "AppleWebKit/537.36 (KHTML, like Gecko) MQQBrowser/7.3 Chrome/37.0.0.0 Mobile Safari/537.36";
        Map fakeAjaxHeaders = new HashMap();
        fakeAjaxHeaders.put("X-Requested-With", "XMLHttpRequest");
        fakeAjaxHeaders.put("Referer", "http://somesite.com/fake_referer");
        CUrl curl = new CUrl("http://httpbin.org/get")
                .opt("-A", mobileUserAgent) // simulate a mobile browser
                .headers(fakeAjaxHeaders)   // simulate an AJAX request
                .header("X-Auth-Token: xxxxxxx"); // other custom header, this might be calculated elsewhere
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例5：多线程并发请求，线程间Cookies相互独立

    public void threadSafeCookies() {
        final CountDownLatch count = new CountDownLatch(3);
        final CUrl[] curls = new CUrl[3];
        for (int i = 3; --i >= 0;) {
            final int idx = i;
            new Thread() {
                public void run() {
                    CUrl curl = curls[idx] = new CUrl("http://httpbin.org/get")
                            .cookie("thread" + idx + "=#" + idx);
                    curl.exec();
                    count.countDown();
                }
            }.start();
        }
        try { count.await(); } catch (Exception ignored) {} // make sure all requests are done
        assertEquals(200, curls[0].getHttpCode());
        assertEquals("thread0=#0", deepGet(curls[0].getStdout(jsonResolver, null), "headers.Cookie"));
        assertEquals("thread1=#1", deepGet(curls[1].getStdout(jsonResolver, null), "headers.Cookie"));
        assertEquals("thread2=#2", deepGet(curls[2].getStdout(jsonResolver, null), "headers.Cookie"));
    }

例6：编程自定义应答解析器，使用JSoup解析HTML

    private CUrl.Resolver htmlResolver = new CUrl.Resolver() {
        @SuppressWarnings("unchecked")
        @Override
        public Document resolve(int httpCode, byte[] responseBody) throws Throwable {
            String html = new String(responseBody, "UTF-8");
            return Jsoup.parse(html);
        }
    };

    public void customResolver() {
        CUrl curl = new CUrl("http://httpbin.org/html");
        Document html = curl.exec(htmlResolver, null);
        assertEquals(200, curl.getHttpCode());
        assertEquals("Herman Melville - Moby-Dick", html.select("h1:first-child").text());
    }

例7：作为命令行工具使用，请求内容参考例4

java -jar java-curl-1.2.0.jar https://httpbin.org/get ^
    -x 127.0.0.1:8888 -k ^
    -A "Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; KNT-AL10 Build/HUAWEIKNT-AL10) AppleWebKit/537.36 (KHTML, like Gecko) MQQBrowser/7.3 Chrome/37.0.0.0 Mobile Safari/537.36" ^
    -H "Referer: http://somesite.com/fake_referer" ^
    -H "X-Requested-With: XMLHttpRequest" ^
    -H "X-Auth-Token: xxxxxxx"