Other instances are discussed in the following thread:
https://lists.gnu.org/r/emacs-devel/2024-02/msg00797.html
* lisp/allout.el (allout-command-prefix): Declare :type as
key-sequence. Mark up key sequences in docstring.
* lisp/auth-source.el (auth-source--decode-octal-string):
* lisp/ffap.el (ffap-search-backward-file-end):
* lisp/gnus/gnus-art.el (gnus-page-delimiter):
* lisp/gnus/nnheader.el (nnheader-strip-cr):
* lisp/proced.el (proced-log):
* lisp/progmodes/idlw-shell.el (idlwave-shell-prompt-pattern):
* lisp/url/url-http.el (url-http-clean-headers):
* lisp/vcursor.el (vcursor-interpret-input): Quote control
characters in docstrings.
It used to be possible to customize 'url-privacy-level' so that the
user's email address was sent along in HTTP requests. Since
'url-privacy-level' is also a blocklist, rather than an allowlist,
this meant that a mere misconfiguration of Emacs risked exposing the
user's email address. This is a serious privacy risk, and it is thus
better if we remove this dangerous feature altogether.
* lisp/url/url-http.el (url-http-create-request): Never send the
user email address.
* lisp/url/url-vars.el (url-personal-mail-address): Make obsolete.
* lisp/url/url-privacy.el (url-setup-privacy-info): Don't set
above obsolete variable.
* doc/misc/url.texi (Customization):
* lisp/url/url-vars.el (url-privacy-level): Update documentation
to reflect the above changes.
* lisp/url/url-util.el (url-display-message): New function.
(url-display-percentage): Make obsolete in favor of
url-display-message.
* lisp/url/url-http.el
(url-http-content-length-after-change-function):
Prefer 'url-display-message' to 'url-display-percentage'.
(url-http-content-length-after-change-function)
(url-http-chunked-encoding-after-change-function):
Remove ineffectual calls to 'url-display-percentage'.
* lisp/net/eww.el (eww--dwim-expand-url): Handle `about: ' URLs.
(bug#56885)
* lisp/url/url-about.el (url-about): Return correct content type
for HTML data.
* lisp/url/url-http.el (url-http--get-referer): Refrain from
looking for a referrer if the lastloc had no host.
* lisp/url/url-http.el (url-http-parse-headers): Output the
headers we receive in the debug output.
* lisp/url/url-vars.el (url-extensions-header): Remove useless header.
* lisp/url/url-http.el (url-http-chunked-last-crlf-missing): Treat
url-http-chunked-last-crlf-missing as any other buffer variable by
declaring and initialising it the same way as the other related
ones (bug#54989).
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el
(url-http-chunked-encoding-after-change-function): Ensure that chunked
encoding is interpreted correctly (bug#54989).
As per [0], the last chunk of 0 bytes is always accompanied by a last
CRLF that signals the end of the message:
chunked-body = *chunk
last-chunk
trailer-part
CRLF
^ this one
chunk = chunk-size [ chunk-ext ] CRLF
chunk-data CRLF
chunk-size = 1*HEXDIG
last-chunk = 1*("0") [ chunk-ext ] CRLF
chunk-data = 1*OCTET ; a sequence of chunk-size octets
`url-http-chunked-encoding-after-change-function' is able to process
(and remove) that terminator IF AVAILABLE in the buffer when
processing the response, however it won't wait for it if it's not yet
there.
In other words:
| Bottom of the response buffer | Bottom of the full response |
| (visible to url-http) | (to be delivered to Emacs) |
| ------------------------------+-----------------------------|
| 0\r\n | 0\r\n |
| | \r\n |
If the last chunk is processed when the bottom of the response buffer
is as above (note that the whole response has not yet been delivered
to Emacs), url-http will call the user callback without waiting for
the final terminator to be read from the socket.
This is normally not an issue when doing one-shot requests, but it's
problematic when the connection is reused immediately. As there are 2
bytes from the request N that have not been dealt with, they'll be
considered as part of the response of the request N+1. On top, it
turns out that when processing the headers of request N+1,
`url-http-wait-for-headers-change-function' will consider the request
a "headerless malformed response" delivering it broken to the caller.
The proposed fix implements a state in which
`url-http-chunked-encoding-after-change-function` properly waits for
the very last element of the message preventing the problem explained
above from happening.
For additional context, this bug was found when debugging
magit/ghub (see [1] for details).
[0] https://datatracker.ietf.org/doc/html/rfc7230#section-4.1
[1] https://github.com/magit/ghub/issues/81
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el (url-http): Factor out url-interactive-p.
* lisp/url/url-auth.el (url-basic-auth):
(url-basic-auth):
(url-digest-prompt-creds): Use it to not query the user.
* lisp/url/url-queue.el (url-queue-start-retrieve): Don't send a
bogus empty Authorization header (bug#54246) -- this triggers
Cloudflare's anti-attack software. Instead rely on
url-request-noninteractive.
* lisp/url/url-vars.el (url-interactive-p): New utility function.
* lisp/url/url-http.el (url-http-handle-authentication): Return t
instead of raising an error, instructing the caller to invoke the
request specific error handler (bug#50511).
* lisp/url/url-http.el (url-http-parse-headers): Parse redirect
URIs more like other web browsers (bug#42382).
RFC 7231 the Location header is defined to carry a URI-reference.
According to RFC 3986 it should be percent-encoded and thus should not
contain spaces. However, there are HTTP server implementation (notably
nginx) that do not do that. This makes Emacs url-http.el behave like
most other HTTP client implementatios. Also remove the stripping of
angle bracket quotes as they are not valid according to the RFCs.
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el (url-http-create-request): Do not recreate
full URL for proxied HTTPS requests.
(url-https-proxy-after-change-function): Do not bind
url-http-proxy to nil before calling url-http-create-request.
(Bug#35969)
* lisp/url/url-http.el (url-http-parse-headers): Narrow the
buffer to the headers at the beginning to make sure
url-handle-content-transfer-encoding uses the correct
headers. (Bug#37023)
It is now called `byte-count-to-string-function', and used instead of
calling `file-size-human-readable' directly where appropriate.
* lisp/files.el (file-size-human-readable-iec): New.
(file-size-function): Rename to byte-count-to-string-function. Better
default value. Eliminate lambda. Better default for custom choice.
Put in group `files'. More descriptive doc string. Move.
(out-of-memory-warning-percentage, warn-maybe-out-of-memory)
(get-free-disk-space):
* lisp/dired.el (dired-number-of-marked-files):
* lisp/url/url-http.el (url-http-simple-after-change-function)
(url-http-content-length-after-change-function):
Use byte-count-to-string-function.
* test/lisp/files-test.el (files-test-file-size-human-readable):
Test file-size-human-readable-iec.
* lisp/url/url-http.el (url-handle-content-transfer-encoding): Modify
the message headers as well as the message body to reflect
decompression.
* lisp/mail/mail-utils.el (mail-fetch-field): Add DELETE argument, to
delete header lines included in the result.
* lisp/url/url-http.el (url-http-handle-authentication): Bail out
if the wrong credentials were passed to the server instead of
inflooping (bug#27022).
To improve readability of strings produced by
`file-size-human-readable', add two optional arguments:
- SPACE, to provide a string (typically a space or non-breaking space)
to put between the number and unit. For compatibility, the default is
an empty string.
- UNIT, a string to use as unit. For compatibility, the default is
"B" in `iec' mode and the empty string otherwise.
Also fix a glitch with small numbers in `iec' mode which caused a
stray "i" in the result.
* lisp/files.el (file-size-human-readable):
Add optional SPACE and UNIT arguments and handle small numbers correctly.
(files--ask-user-about-large-file, warn-maybe-out-of-memory):
Call with `iec' and space.
* test/lisp/files-tests.el (files-test-file-size-human-readable): New test.
* lisp/url/url-http.el (url-http-simple-after-change-function)
(url-http-content-length-after-change-function): Call with `iec' and space.
* etc/NEWS (Lisp Changes): Mention the change.
Restore lines saying "Maintainer: emacs-devel@gnu.org" when there is
no special maintainer for a file. Although this wasn't documented
it was common practice and removing the lines didn't have consensus.
This is a different fix for bug#34909, which should also fix bug#35739.
Our downloading code used to automatically decode the result according
to the usual heuristics for files. This caused problems when we later
needed to save the data in a file that needed to be byte-for-byte
equal to the original in order to pass the signature verification,
especially because we didn't keep track of which coding-system was
used to decode the data.
(package--unless-error): New macro extracted from
package--with-response-buffer-1, so that we can specify edebug and
indent specs.
(package--with-response-buffer-1): Use it. More importantly, change
code so it runs `body` in a unibyte buffer with undecoded data.
(package--download-one-archive): Don't encode with utf-8 since the data
is not decoded yet.
(describe-package-1): Explicitly decode the readem.txt files here.
* lisp/url/url-handlers.el (url-insert-file-contents): Use it.
(url-insert): Don't decode if buffer is unibyte.
* lisp/url/url-http.el (url-http--insert-file-helper): New function,
extracted from url-insert-file-contents.
* lisp/url/url-http.el
(url-http-chunked-encoding-after-change-function): Ensure that we
parse the entire initial chunked header as the length (bug#35658).
