* lisp/url/url-http.el (url-http-chunked-last-crlf-missing): Treat
url-http-chunked-last-crlf-missing as any other buffer variable by
declaring and initialising it the same way as the other related
ones (bug#54989).
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el
(url-http-chunked-encoding-after-change-function): Ensure that chunked
encoding is interpreted correctly (bug#54989).
As per [0], the last chunk of 0 bytes is always accompanied by a last
CRLF that signals the end of the message:
chunked-body = *chunk
last-chunk
trailer-part
CRLF
^ this one
chunk = chunk-size [ chunk-ext ] CRLF
chunk-data CRLF
chunk-size = 1*HEXDIG
last-chunk = 1*("0") [ chunk-ext ] CRLF
chunk-data = 1*OCTET ; a sequence of chunk-size octets
`url-http-chunked-encoding-after-change-function' is able to process
(and remove) that terminator IF AVAILABLE in the buffer when
processing the response, however it won't wait for it if it's not yet
there.
In other words:
| Bottom of the response buffer | Bottom of the full response |
| (visible to url-http) | (to be delivered to Emacs) |
| ------------------------------+-----------------------------|
| 0\r\n | 0\r\n |
| | \r\n |
If the last chunk is processed when the bottom of the response buffer
is as above (note that the whole response has not yet been delivered
to Emacs), url-http will call the user callback without waiting for
the final terminator to be read from the socket.
This is normally not an issue when doing one-shot requests, but it's
problematic when the connection is reused immediately. As there are 2
bytes from the request N that have not been dealt with, they'll be
considered as part of the response of the request N+1. On top, it
turns out that when processing the headers of request N+1,
`url-http-wait-for-headers-change-function' will consider the request
a "headerless malformed response" delivering it broken to the caller.
The proposed fix implements a state in which
`url-http-chunked-encoding-after-change-function` properly waits for
the very last element of the message preventing the problem explained
above from happening.
For additional context, this bug was found when debugging
magit/ghub (see [1] for details).
[0] https://datatracker.ietf.org/doc/html/rfc7230#section-4.1
[1] https://github.com/magit/ghub/issues/81
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el (url-http): Factor out url-interactive-p.
* lisp/url/url-auth.el (url-basic-auth):
(url-basic-auth):
(url-digest-prompt-creds): Use it to not query the user.
* lisp/url/url-queue.el (url-queue-start-retrieve): Don't send a
bogus empty Authorization header (bug#54246) -- this triggers
Cloudflare's anti-attack software. Instead rely on
url-request-noninteractive.
* lisp/url/url-vars.el (url-interactive-p): New utility function.
* lisp/url/url-http.el (url-http-handle-authentication): Return t
instead of raising an error, instructing the caller to invoke the
request specific error handler (bug#50511).
* lisp/url/url-http.el (url-http-parse-headers): Parse redirect
URIs more like other web browsers (bug#42382).
RFC 7231 the Location header is defined to carry a URI-reference.
According to RFC 3986 it should be percent-encoded and thus should not
contain spaces. However, there are HTTP server implementation (notably
nginx) that do not do that. This makes Emacs url-http.el behave like
most other HTTP client implementatios. Also remove the stripping of
angle bracket quotes as they are not valid according to the RFCs.
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el (url-http-create-request): Do not recreate
full URL for proxied HTTPS requests.
(url-https-proxy-after-change-function): Do not bind
url-http-proxy to nil before calling url-http-create-request.
(Bug#35969)
* lisp/url/url-http.el (url-http-parse-headers): Narrow the
buffer to the headers at the beginning to make sure
url-handle-content-transfer-encoding uses the correct
headers. (Bug#37023)
It is now called `byte-count-to-string-function', and used instead of
calling `file-size-human-readable' directly where appropriate.
* lisp/files.el (file-size-human-readable-iec): New.
(file-size-function): Rename to byte-count-to-string-function. Better
default value. Eliminate lambda. Better default for custom choice.
Put in group `files'. More descriptive doc string. Move.
(out-of-memory-warning-percentage, warn-maybe-out-of-memory)
(get-free-disk-space):
* lisp/dired.el (dired-number-of-marked-files):
* lisp/url/url-http.el (url-http-simple-after-change-function)
(url-http-content-length-after-change-function):
Use byte-count-to-string-function.
* test/lisp/files-test.el (files-test-file-size-human-readable):
Test file-size-human-readable-iec.
* lisp/url/url-http.el (url-handle-content-transfer-encoding): Modify
the message headers as well as the message body to reflect
decompression.
* lisp/mail/mail-utils.el (mail-fetch-field): Add DELETE argument, to
delete header lines included in the result.
* lisp/url/url-http.el (url-http-handle-authentication): Bail out
if the wrong credentials were passed to the server instead of
inflooping (bug#27022).
To improve readability of strings produced by
`file-size-human-readable', add two optional arguments:
- SPACE, to provide a string (typically a space or non-breaking space)
to put between the number and unit. For compatibility, the default is
an empty string.
- UNIT, a string to use as unit. For compatibility, the default is
"B" in `iec' mode and the empty string otherwise.
Also fix a glitch with small numbers in `iec' mode which caused a
stray "i" in the result.
* lisp/files.el (file-size-human-readable):
Add optional SPACE and UNIT arguments and handle small numbers correctly.
(files--ask-user-about-large-file, warn-maybe-out-of-memory):
Call with `iec' and space.
* test/lisp/files-tests.el (files-test-file-size-human-readable): New test.
* lisp/url/url-http.el (url-http-simple-after-change-function)
(url-http-content-length-after-change-function): Call with `iec' and space.
* etc/NEWS (Lisp Changes): Mention the change.
Restore lines saying "Maintainer: emacs-devel@gnu.org" when there is
no special maintainer for a file. Although this wasn't documented
it was common practice and removing the lines didn't have consensus.
This is a different fix for bug#34909, which should also fix bug#35739.
Our downloading code used to automatically decode the result according
to the usual heuristics for files. This caused problems when we later
needed to save the data in a file that needed to be byte-for-byte
equal to the original in order to pass the signature verification,
especially because we didn't keep track of which coding-system was
used to decode the data.
(package--unless-error): New macro extracted from
package--with-response-buffer-1, so that we can specify edebug and
indent specs.
(package--with-response-buffer-1): Use it. More importantly, change
code so it runs `body` in a unibyte buffer with undecoded data.
(package--download-one-archive): Don't encode with utf-8 since the data
is not decoded yet.
(describe-package-1): Explicitly decode the readem.txt files here.
* lisp/url/url-handlers.el (url-insert-file-contents): Use it.
(url-insert): Don't decode if buffer is unibyte.
* lisp/url/url-http.el (url-http--insert-file-helper): New function,
extracted from url-insert-file-contents.
* lisp/url/url-http.el
(url-http-chunked-encoding-after-change-function): Ensure that we
parse the entire initial chunked header as the length (bug#35658).
* lisp/url/url-http.el (url-http-debug):
* lisp/url/url-util.el (url-debug): Don't signal an error if
quit-flag is non-nil, but not t. This could happen because
some unrelated code is running inside while-no-input.
(Bug#34763)
* lisp/obsolete/tls.el: Moved from lisp/net/tls.el.
* lisp/gnus/nnimap.el:
* lisp/url/url-http.el: Don't require tls, since it's obsolete.
* lisp/net/network-stream.el: Only require tls if we actually try to
use it (i.e., when (gnutls-available-p) returns nil). Declare some
functions to fix compilation warnings.
* lisp/obsolete/starttls.el: Moved from lisp/net/starttls.el.
* lisp/net/sieve-manage.el:
* lisp/net/network-stream.el: Don't require `starttls' at the
top-level, declare the variables and functions used instead.
(network-stream-open-starttls): Only require `starttls' if
needed (i.e., gnutls-available-p fails).
* etc/NEWS: Announce obsoletion.
* lisp/net/gnutls.el (open-gnutls-stream): IDNA-encode hostnames
before passing them on to gnutls for verification.
* lisp/net/network-stream.el (network-stream-open-starttls): Ditto.
* lisp/url/url-http.el (url-http--get-referer): Be IDNA-aware.
(url-http-create-request): Don't de-Unicodify host names, because
they may be IDNA names (that are later encoded).
* lisp/url/url-util.el (url-domain): Be IDNA-aware when doing
domain name computations.
