If a TLS handshake is attempted before the completion of an
asynchronous TCP connection has been ascertained, our local state will
not be set up correctly for further progress and the sentinel "open"
event will never be sent. This can occur if sufficient time passes
after the initiation of an async TCP connection so that by the time
`wait_reading_process_output` is called, the connection has already
been established on the TCP level.
This somewhat timing-sensitive bug has plagued HTTPS connections on
some platforms, notably macOS, for a long time (bug#49449).
* src/process.c (wait_reading_process_output): Gate the TLS handshake
by the NON_BLOCKING_CONNECT_FD flag. The flag will be cleared as soon
as the TCP socket is found to be writable.
* test/src/process-tests.el (process-async-https-with-delay):
New test.
* test/src/process-tests.el
(process-tests/fd-setsize-no-crash/make-network-process): Skip
test on Cygwin to avoid hang due to connect/accept handshake.
(Bug#49496)
Bug#46641
The network-stream-tests actually work fine when the local machine has
no IP at all, but cause a crash in the GnuTLS library when there is an
IP configured but the interface is down.
* test/lisp/net/network-stream-tests.el (internet-is-working): New
defvar, checks if we can resolve "google.com".
(connect-to-tls-ipv4-nowait, connect-to-tls-ipv6-nowait)
(open-network-stream-tls-nowait, open-gnutls-stream-new-api-nowait)
(open-gnutls-stream-old-api-nowait): Use it to check for working
Internet access.
* test/src/process-tests.el (internet-is-working): New defvar, checks
if we can resolve "google.com".
(lookup-family-specification, lookup-unicode-domains)
(unibyte-domain-name, lookup-google, non-existent-lookup-failure): Use
it to check for working Internet access.
* test/src/process-tests.el (process-tests/multiple-threads-waiting):
Also check that 'thread-join' and 'thread-last-error' return the
expected errors.
This unit test tests that we can call 'accept-process-output' in
parallel from multiple threads.
* test/src/process-tests.el (process-tests/multiple-threads-waiting):
New unit test.
That way, the tests only depend on Emacs, and not on utilities that
might not be available during test time.
* test/src/process-tests.el (process-tests--eval)
(process-tests--emacs-command, process-tests--emacs-binary)
(process-tests--dump-file)
(process-tests--usable-file-for-reinvoke): New helper functions.
(process-tests/sentinel-called)
(process-tests/sentinel-with-multiple-processes): Use them.
When we're trying to notify a process about a status change, we need
to ignore the SIGCHLD pipe temporarily, otherwise the code would
likely not run into the timeout case that's necessary for a status
change to happen.
* src/process.c (wait_reading_process_output): Ignore the SIGCHLD pipe
when notifying a process about a status change.
* test/src/process-tests.el (process-tests/sentinel-called)
(process-tests/sentinel-with-multiple-processes): New unit tests.
If we receive and handle a SIGCHLD signal for a process while waiting
for that process, 'pselect' might never return. Instead, we have to
explicitly 'pselect' that the process status has changed. We do this
by writing to a pipe in the SIGCHLD handler and having
'wait_reading_process_output' select on it.
* src/process.c (child_signal_init): New helper function to create a
pipe for SIGCHLD notifications.
(child_signal_read, child_signal_notify): New helper functions to
read from/write to the child signal pipe.
(create_process): Initialize the child signal pipe on first use.
(handle_child_signal): Notify waiters that a process status has
changed.
(wait_reading_process_output): Make sure that we also catch
SIGCHLD/process status changes.
* test/src/process-tests.el
(process-tests/fd-setsize-no-crash/make-process): Remove workaround,
which is no longer needed.
* test/src/process-tests.el (ipv6-is-available): New function for
checking whether we have a globally routable IPv6 prefix assigned.
(lookup-family-specification): Use 'ipv6-is-available' to check for
IPv6. Use 'localhost' instead of 'google.com' to test
'network-lookup-address-info' API.
(lookup-google): Use 'ipv6-is-available' to check for
IPv6.
* test/lisp/net/nsm-tests.el (nsm-ipv6-is-available): Rename to
'ipv6-is-available', make identical to the one in
test/src/process-tests.el.
The test 'process-tests/fd-setsize-no-crash/make-process' used to call
'sleep' to ensure that enough processes are live to trigger a
FD_SETSIZE overflow. However, we can just call 'cat' instead and
close standard input when done. That way, we only wait as long as
needed.
* process-tests.el
(process-tests/fd-setsize-no-crash/make-process): Invoke 'cat' instead
of 'sleep'. Close standard input to exit the 'cat' processes.
The 'process-tty-name' already provides the TTY name, we don't have
interrogate the TTY host.
* test/src/process-tests.el
(process-tests/fd-setsize-no-crash/make-serial-process): Use
'process-tty-name' instead of having the TTY host print its TTY
name. Check whether TTY names are unique.
(process-tests--new-pty, process-tests--with-temp-file): Remove;
no longer used.
This works even if another thread or process resets the resource limit
for open file descriptors, e.g., using 'prlimit' on GNU/Linux.
* src/process.c (create_process, create_pty, Fmake_pipe_process)
(Fmake_serial_process, connect_network_socket)
(server_accept_connection): Limit file descriptors to FD_SETSIZE.
* test/src/process-tests.el (process-tests--with-raised-rlimit): New
helper macro.
(process-tests--fd-setsize-test): Rename from
'process-tests--with-many-pipes'. Increase resource limit during test
if possible.
(process-tests/fd-setsize-no-crash/make-process)
(process-tests/fd-setsize-no-crash/make-pipe-process)
(process-tests/fd-setsize-no-crash/make-network-process)
(process-tests--new-pty): Rename callers.
* test/src/process-tests.el
(process-tests/fd-setsize-no-crash/make-process): Allow special exit
codes that can happen if terminal setup fails in the child process.
Instead of trying to generate the right number of processes,
pre-create lots of unused pipe processes until creation fails. Extend
the tests to the 'pty' connection type and other kinds of process
objects.
* test/src/process-tests.el (process-tests--ignore-EMFILE)
(process-tests--with-buffers, process-tests--with-processes)
(process-tests--with-many-pipes, process-tests--with-temp-file)
(process-tests--with-temp-directory): New helper macros.
(process-tests/fd-setsize-no-crash/make-process): Renamed from
'process-tests/fd-setsize-no-crash'. Fail on timeout. Also test the
'pty' connection type. Pre-create lots of pipe processes so we reach
the FD_SETSIZE limit faster. Ignore EMFILE more precisely, if
possible.
(process-tests/fd-setsize-no-crash/make-pipe-process)
(process-tests/fd-setsize-no-crash/make-network-process)
(process-tests/fd-setsize-no-crash/make-serial-process): New tests
that test FD_SETSIZE limits for other kinds of processes.
(process-tests--EMFILE-message): New helper function and cache
variable.
(process-tests--new-pty): New helper function.
* src/w32.c (pipe2): When forcibly closing pipe handles due to
overflow of FD_SETSIZE, set the handles to -1, to avoid assertion
violations in emacs_close.
* test/src/process-tests.el (process-tests/fd-setsize-no-crash):
No need to skip this test anymore.
* test/src/process-tests.el (process-tests/fd-setsize-no-crash):
Skip this test on windows-nt systems, as we cannot use more than
FD_SETSIZE file descriptors there: if we try, we crash.
* src/process.c (Fmake_network_process) [AF_INET6]: Only build ::1
localhost when IPv6 is supported.
(Fnetwork_lookup_address_info) [AF_INET6]: Move check for Qipv6 inside
ifdef, since its definition depends on AF_INET6. Don't return IPv6
addresses when they're not supported.
* test/src/process-tests.el (lookup-family-specification,
lookup-google): Only do IPv6 lookup if IPv6 is supported.
* test/src/process-tests.el (set-process-filter-t): Don't assume
subprocess output will come in a single chunk, keep waiting for more
data until next "prompt" is read from subprocess.
* test/src/process-tests.el: (lookup-family-specification,
lookup-unicode-domains, unibyte-domain-name, lookup-google,
non-existent-lookup-failure): Skip on Hydra, which doesn't have DNS.
Fix buggy test condition. (Bug#37165)
* src/process.c (Fset_process_filter): Call add_process_read_fd
according to the state of process filter before it's updated. This
restores the correct functioning as it was before 2016-02-16 "Allow
setting the filter masks later". Inline the set_process_filter_masks
call instead of fixing it that function, because it is also called
from connect_network_socket, and we don't want to change the behavior
of that function so close to release.
* test/src/process-tests.el (set-process-filter-t): New test.
This has never worked and caused issues such as Bug#30460.
* src/process.c (Fmake_process): Don't accept :stop key any more.
(syms_of_process): Define needed symbol 'null'.
* test/src/process-tests.el (make-process/stop): New unit test.
* doc/lispref/processes.texi (Asynchronous Processes): Remove :stop
key from manual.
Since child_setup() is called between block_input()...unblock_input(),
when an error is signaled the Lisp debugger is prevented from
starting. Therefore, let the callers signal the error instead (which
they already do for non-w32 platforms, just the error message needs an
update).
* src/callproc.c (child_setup) [WINDOWSNT]: Don't call
report_file_error here.
(call_process) [WINDOWNT]:
* src/process.c (create_process) [WINDOWSNT]: Call report_file_errno
here instead, after the unblock_input() call, same as for !WINDOWSNT.
* src/lisp.h (CHILD_SETUP_ERROR_DESC): New preprocessor define. Flip
the containing ifndef DOS_NT branches so that it's ifdef DOS_NT.
* src/eval.c (when_entered_debugger): Remove.
(syms_of_eval) <internal-when-entered-debugger>: Define it as a Lisp
integer variable instead.
(maybe_call_debugger): Update comment.
* test/src/process-tests.el (make-process-w32-debug-spawn-error):
* test/src/callproc-tests.el (call-process-w32-debug-spawn-error): New
tests.
* src/process.c (Fmake_process): Add new keyword argument
':file-handler'.
(syms_of_process) <make-process, :file-handler>: Define new symbols.
* lisp/files.el (file-name-non-special): Add support for
'make-process'.
* test/src/process-tests.el (make-process/file-handler/found)
(make-process/file-handler/not-found)
(make-process/file-handler/disable): New unit tests.
(process-tests--file-handler): New helper function.
* test/lisp/files-tests.el
(files-tests-file-name-non-special-make-process): New unit test.
* doc/lispref/files.texi (Magic File Names): Document that
'make-process' can invoke file name handlers.
* doc/lispref/processes.texi (Asynchronous Processes): Document
':file-handlers' argument to 'make-process'.
* etc/NEWS (Lisp Changes in Emacs 27.1): Mention new
:file-handler argument for 'make-process'.
* src/process.c (network_lookup_address_info_1): Only check multibyte
hostnames for pure-ASCII.
* test/src/process-tests.el (unibyte-domain-name): Test unibyte domain
names with network-lookup-address-info.
* test/src/process-tests.el (lookup-family-specification): Test
network-lookup-address-info api.
(lookup-unicode-domains): Test that unicode domains fail.
(lookup-google): Test that normal lookups succeed.
(non-existent-lookup-failure): Check that known non-existent
domains fail.
* test/src/process-tests.el (process-tests--mixable): New function.
(make-process/mix-stderr): Don’t assume stdout is merged before
stderr. POSIX does not require this, and the assumption failed to
hold on my Fedora 28 platform. See Bug#31214.
