* lisp/url/url-irc.el (url-irc-function): Change signature of function
interface to expect a final "scheme" argument, such as "ircs".
(url-irc): Call `url-irc-function' with new positional argument, the
scheme extracted via `url-type' from the input URL.
(url-irc-erc, url-irc-rcirc, url-irc-zenirc): Accept a URL scheme as a
sixth positional arg.
(url-ircs-default-port, url-ircs): Add new autoloaded constant and
alias for `url-scheme-get-property' to recognize. Do this to avoid
having to add another file.
* lisp/net/browse-url.el (browse-url-irc-function): Add new option.
(browse-url--irc): Add new function to call `browse-url-irc-function'.
(browse-url-default-handlers): Add "irc://" entry.
(browse-url-irc): Add new function to serve as general handler for
"irc://" URLS. Accept trailing variadic args to accommodate
non-browse-url interfaces as well.
* test/lisp/net/browse-url-tests.el
(browse-url-tests-select-handler-irc): Add test for "irc://" URL
pattern.
* etc/NEWS: Mention select browse-url and url-irc
changes. (Bug#56514.)
Binding it causes errors when some of the enclosed code ends up
loading a `.el` file, and it is unnecessary since the rest of
the code (in `url-file-asynch-callback`) is already careful to
use `insert-file-contents-literally`.
* lisp/gnus/gnus-group.el (gnus-read-ephemeral-bug-group): Don't
bind coding system variables, because that leads to loading other
files (like cookie files) using that coding system (bug#58227).
* lisp/url/url-handlers.el (url-insert-file-literally): New function.
(url-insert): Allow forcing no-decode.
* lisp/url/url.el (url-retrieve-synchronously): Don't kill the
process buffer unless we made one (bug#58218). This makes HTTP
free connection marking work on premature timeouts.
* lisp/url/url-util.el (url-display-message): New function.
(url-display-percentage): Make obsolete in favor of
url-display-message.
* lisp/url/url-http.el
(url-http-content-length-after-change-function):
Prefer 'url-display-message' to 'url-display-percentage'.
(url-http-content-length-after-change-function)
(url-http-chunked-encoding-after-change-function):
Remove ineffectual calls to 'url-display-percentage'.
* lisp/net/eww.el (eww--dwim-expand-url): Handle `about: ' URLs.
(bug#56885)
* lisp/url/url-about.el (url-about): Return correct content type
for HTML data.
* lisp/url/url-http.el (url-http--get-referer): Refrain from
looking for a referrer if the lastloc had no host.
* lisp/url/url-util.el (url-eat-trailing-space): Redefine as
obsolete function alias for 'string-trim-right'.
(url-strip-leading-spaces): Redefine as obsolete function alias
for 'string-trim-left'.
The w3 package was removed from GNU ELPA in 2020 as it doesn't run on
a recent Emacs, and development had stopped over a decade before that.
If anyone wants to revive the w3 package, they should look this all
over, but it doesn't make sense for us to maintain this support code.
Ref: https://debbugs.gnu.org/25395
* lisp/net/browse-url.el (browse-url-w3): Make obsolete.
(browse-url--browser-defcustom-type)
(browse-url-default-browser):
* lisp/ffap.el (ffap-url-at-point, ffap-file-at-point):
* lisp/net/newst-plainview.el (newsticker--buffer-do-insert-text):
* lisp/net/newst-reader.el (newsticker-html-renderer)
(newsticker-show-news):
* lisp/net/newst-treeview.el (newsticker--treeview-render-text):
* lisp/org/ol.el (org-store-link):
* lisp/url/url.el (url-retrieve): Remove w3 support code and related
documentation and comments. (Bug#56435)
* test/lisp/net/browse-url-tests.el
(browse-url-tests-browser-kind): Adjust test for above changes.
* etc/TODO: Remove TODO to install W3.
* doc/misc/org.org (Handling Links): Don't mention W3.
* lisp/msb.el (msb--few-menus, msb--very-many-menus): Check for
eww-mode instead of w3-mode.
* lisp/url/url-cookie.el (url-cookie-write-file):
* lisp/international/titdic-cnv.el (tit-process-header):
* lisp/international/ja-dic-cnv.el (skkdic-convert):
* lisp/international/emoji.el (emoji--generate-file):
* lisp/emacs-lisp/loaddefs-gen.el (loaddefs-generate--rubric)
* admin/unidata/unidata-gen.el (unidata-gen-file)
(unidata-gen-charprop): Use the new functions.
* lisp/emacs-lisp/generate-file.el: New file to provide
convenience functions for generated files. It's not always
trivial to know which parts of the trailer that has to be
obfuscated to avoid not getting byte-compiled etc, and some parts
of the headers/trailers are usually forgotten when hand-coding
these.
* lisp/url/url-http.el (url-http-parse-headers): Output the
headers we receive in the debug output.
* lisp/url/url-vars.el (url-extensions-header): Remove useless header.
* lisp/url/url-http.el (url-http-chunked-last-crlf-missing): Treat
url-http-chunked-last-crlf-missing as any other buffer variable by
declaring and initialising it the same way as the other related
ones (bug#54989).
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el
(url-http-chunked-encoding-after-change-function): Ensure that chunked
encoding is interpreted correctly (bug#54989).
As per [0], the last chunk of 0 bytes is always accompanied by a last
CRLF that signals the end of the message:
chunked-body = *chunk
last-chunk
trailer-part
CRLF
^ this one
chunk = chunk-size [ chunk-ext ] CRLF
chunk-data CRLF
chunk-size = 1*HEXDIG
last-chunk = 1*("0") [ chunk-ext ] CRLF
chunk-data = 1*OCTET ; a sequence of chunk-size octets
`url-http-chunked-encoding-after-change-function' is able to process
(and remove) that terminator IF AVAILABLE in the buffer when
processing the response, however it won't wait for it if it's not yet
there.
In other words:
| Bottom of the response buffer | Bottom of the full response |
| (visible to url-http) | (to be delivered to Emacs) |
| ------------------------------+-----------------------------|
| 0\r\n | 0\r\n |
| | \r\n |
If the last chunk is processed when the bottom of the response buffer
is as above (note that the whole response has not yet been delivered
to Emacs), url-http will call the user callback without waiting for
the final terminator to be read from the socket.
This is normally not an issue when doing one-shot requests, but it's
problematic when the connection is reused immediately. As there are 2
bytes from the request N that have not been dealt with, they'll be
considered as part of the response of the request N+1. On top, it
turns out that when processing the headers of request N+1,
`url-http-wait-for-headers-change-function' will consider the request
a "headerless malformed response" delivering it broken to the caller.
The proposed fix implements a state in which
`url-http-chunked-encoding-after-change-function` properly waits for
the very last element of the message preventing the problem explained
above from happening.
For additional context, this bug was found when debugging
magit/ghub (see [1] for details).
[0] https://datatracker.ietf.org/doc/html/rfc7230#section-4.1
[1] https://github.com/magit/ghub/issues/81
Copyright-paperwork-exempt: yes
* lisp/url/url-http.el (url-http): Factor out url-interactive-p.
* lisp/url/url-auth.el (url-basic-auth):
(url-basic-auth):
(url-digest-prompt-creds): Use it to not query the user.
* lisp/url/url-queue.el (url-queue-start-retrieve): Don't send a
bogus empty Authorization header (bug#54246) -- this triggers
Cloudflare's anti-attack software. Instead rely on
url-request-noninteractive.
* lisp/url/url-vars.el (url-interactive-p): New utility function.
* lisp/net/eww.el (eww-open-file): Use it.
(eww-browse-url): Ditto.
* lisp/net/shr.el (shr-expand-url): Allow loading relative Tramp
files if we're reading from a file:// document.
* lisp/url/url-file.el (url-allow-non-local-files): New user option
(bug#40425).
