Improve Dired handling of file names with newlines (bug#79528)

* doc/emacs/dired.texi (Dired Enter): Update documentation of
Dired's display and handling of file names that contain newlines.
Document new user option 'dired-auto-toggle-b-switch'.

* etc/NEWS: Announce new warning and new user option.

* lisp/dired.el (dired-auto-toggle-b-switch): New user option.
(dired-internal-noselect): Pop up a warning if the Dired listing
displays a literal newline.
(dired-switches-escape-p): Take the 'ls' switch
'--quoting-style=escape' into account.
(dired-mode): Add 'dired--toggle-b-switch' to 'post-command-hook'.
(dired-move-to-end-of-filename): When the Dired listing includes a
file name containing a newline, this can result in no change in
the 'dired-filename' text property on the last file name, so in
this case take the position just before the final newline as the
end of the last file name to prevent a wrong-type-argument error.
(dired--filename-with-newline-p, dired--remove-b-switch)
(dired--toggle-b-switch, dired--set-auto-toggle-b-switch)
(dired--display-filename-with-newline-warning): New functions.

doc/emacs/dired.texi

@@ -136,15 +136,57 @@ options (that is, single characters) requiring no arguments, and long
 options (starting with @samp{--}) whose arguments are specified with
 @samp{=}.
 
-  Dired does not handle files that have names with embedded newline
+You can declare @code{dired-listing-switches} as a connection-local
-characters well.  If you have many such files, you may consider adding
+variable in order to adjust its value to match what a remote system
-@samp{-b} to @code{dired-listing-switches}.  This will quote all
+expects (@pxref{Connection Variables}).
-special characters and allow Dired to handle them better.  (You can
-also use the @kbd{C-u C-x d} command to add @samp{-b} temporarily.)
 
-@code{dired-listing-switches} can be declared as connection-local
+@cindex file names with newline character in Dired
-variable to adjust it to match what a remote system expects
+@cindex newline character in file names in Dired
-(@pxref{Connection Variables}).
+@anchor{File names with newline}
+  When a file name contains a newline character, Dired displays it by
+default as a literal newline, so the display of this file name occupies
+more than one line in the Dired buffer.  If you invoke a Dired operation
+on such a file listing, in many cases it will fail and signal an error.
+For this reason, when Dired displays a file name containing a literal
+newline, Emacs recognizes this and automatically pops up a buffer with
+an informative warning.  For such file names, Dired offers an
+alternative display, using the @command{ls} switch @samp{-b}, in which
+newline characters are represented by @samp{\n} and the Dired listing of
+the file occupies one line as usual, so you can execute all applicable
+Dired operations on it.@footnote{Note that with the @samp{-b} switch
+Dired displays tab characters in file names as @samp{\t} and escapes
+other control characters and also spaces in file names with @samp{\}.}
+
+  Emacs provides two different ways to make Dired use the @samp{-b}
+switch:
+
+@itemize @bullet
+@item
+You can add @samp{-b} to @code{dired-listing-switches} before invoking
+@code{dired}.  Since this variable is a user option, you can alter its
+value persistently either by using the Customization interface
+(@pxref{Saving Customizations}) or by using the @code{setopt} macro in
+your initialization file (@pxref{Examining}).@footnote{If
+@code{dired-listing-switches} contains @samp{-b} when you invoke dired
+on a directory containing a file name with a newline, the newline is
+displayed as @samp{\n}, so Emacs will not pop up a warning.}  You can
+also add @samp{-b} just for the next @code{dired} invocation by typing
+@kbd{C-u C-x d}.
+
+@item
+@vindex dired-auto-toggle-b-switch
+If you enable the user option @code{dired-auto-toggle-b-switch}, then
+when you visit a directory containing a file whose name has a newline,
+Emacs will automatically add the @samp{-b} switch and redisplay the
+directory in Dired to show @samp{\n} in the file name.  If you edit the
+file name and remove the @samp{\n} character, then on completing the
+edit Emacs automatically removes the @samp{-b} switch and redisplays the
+Dired buffer, so that file names with tab or space characters now show
+literal spaces without a backslash.  If you enable or disable
+@code{dired-auto-toggle-b-switch} after visiting a directory containing
+a file name with a newline, Emacs will add or remove the @samp{-b}
+switch as appropriate and automatically redisplay the Dired buffer.
+@end itemize
 
 @vindex dired-switches-in-mode-line
   Dired displays in the mode line an indication of what were the

etc/NEWS

@@ -2042,6 +2042,24 @@ name of the directory now reverts the Dired buffer.
 With a new value of the prefix argument (1), this command copies file
 names relative to the root directory of the current project.
 
++++
+*** Warning when Dired displays a file name with a literal newline.
+On visiting a directory that contains a file whose name has a newline,
+and Dired displays that character as a literal newline, Emacs now
+automatically pops up a buffer warning that such a display can be
+problematic for Dired and showing a way to change the display to use the
+unproblematic character '\n'.
+
++++
+*** New user option 'dired-auto-toggle-b-switch'.
+When this user option is non-nil and 'dired-listing-switches' does not
+include the '-b' switch, then on visiting a directory containing a file
+whose name has a newline, Emacs automatically adds the '-b' switch and
+redisplays the directory in Dired to show '\n' in the file name instead
+of a literal newline.  This prevents executing many Dired operations on
+such a file from failing and signaling an error.  The default value of
+this user option is nil.
+
 ** Grep
 
 +++

lisp/dired.el

@@ -550,6 +550,16 @@ displayed instead."
   :group 'dired
   :version "30.1")
 
+(defcustom dired-auto-toggle-b-switch nil
+  "Whether to automatically add or remove the `b' switch.
+If non-nil, the function `dired--toggle-b-switch' (which see) is added
+to `post-command-hook' in Dired mode."
+  :type 'boolean
+  :group 'dired
+  :initialize #'custom-initialize-default
+  :set #'dired--set-auto-toggle-b-switch
+  :version "31.1")
+
 
 ;;; Internal variables
 
@@ -1437,6 +1447,16 @@ The return value is the target column for the file names."
       (dired-initial-position dirname))
     (when (consp dired-directory)
       (dired--align-all-files))
+    ;; Pop up a warning if the Dired listing displays a literal newline.
+    ;; We do this here in order to get the warning not only when
+    ;; interactively invoking `dired' on a directory, but also e.g. when
+    ;; passing the directory name as a command line argument when
+    ;; starting Emacs from the shell.
+    (unless (or dired-auto-toggle-b-switch
+                (dired-switches-escape-p dired-listing-switches)
+                (dired-switches-escape-p dired-actual-switches))
+      (when (dired--filename-with-newline-p)
+        (dired--display-filename-with-newline-warning buffer)))
     (set-buffer old-buf)
     buffer))
 
@@ -1699,7 +1719,7 @@ BEG..END is the line where the file info is located."
 (defun dired-switches-escape-p (switches)
   "Return non-nil if the string SWITCHES contains -b or --escape."
   ;; Do not match things like "--block-size" that happen to contain "b".
-  (dired-check-switches switches "b" "escape"))
+  (dired-check-switches switches "b" "\\(quoting-style=\\)?escape"))
 
 (defun dired-switches-recursive-p (switches)
   "Return non-nil if the string SWITCHES contains -R or --recursive."
@@ -2855,6 +2875,8 @@ Keybindings:
   (add-hook 'file-name-at-point-functions #'dired-file-name-at-point nil t)
   (add-hook 'isearch-mode-hook #'dired-isearch-filenames-setup nil t)
   (add-hook 'context-menu-functions 'dired-context-menu 5 t)
+  (when dired-auto-toggle-b-switch
+    (add-hook 'post-command-hook #'dired--toggle-b-switch nil t))
   (run-mode-hooks 'dired-mode-hook))
 
 
@@ -3439,7 +3461,14 @@ If EOL, it should be an position to use instead of
   ;; On failure, signals an error (with non-nil NO-ERROR just returns nil).
   ;; This is the UNIX version.
   (if (get-text-property (point) 'dired-filename)
-      (goto-char (next-single-property-change (point) 'dired-filename))
+      (goto-char (or (next-single-property-change (point) 'dired-filename)
+                     ;; No property change can happen on or before the
+                     ;; last file name in the Dired listing when there
+                     ;; is at least one prior file name containing a
+                     ;; newline.  To prevent an error in this case we
+                     ;; take the position just before the final newline
+                     ;; as the end of the last file name (bug#79528).
+                     (1- (point-max))))
     (let ((opoint (point))
           (used-F (dired-check-switches dired-actual-switches "F" "classify"))
           (eol (line-end-position))
@@ -3973,6 +4002,98 @@ Considers buffers closer to the car of `buffer-list' to be more recent."
        (memq buffer1 (buffer-list))
        (not (memq buffer1 (memq buffer2 (buffer-list))))))
 
+(defun dired--filename-with-newline-p ()
+  "Check if a file name in this directory has a newline.
+Return non-nil if at least one file name in this directory contains
+either a literal newline or the string \"\\n\")."
+  (save-excursion
+    (goto-char (point-min))
+    (catch 'found
+      (while (not (eobp))
+        (when (dired-move-to-filename)
+          (let ((fn (buffer-substring-no-properties
+                     (point) (dired-move-to-end-of-filename))))
+            (when (or (memq 10 (seq-into fn 'list))
+                      (string-search "\\n" fn))
+              (throw 'found t))))
+        (forward-line)))))
+
+(defun dired--remove-b-switch ()
+  "Remove all variants of the `b' switch from `dired-actual-switches'.
+This removes not only all occurrences of the short form `-b' but also
+the long forms `--escape' and `--quoting-style=escape'."
+  (let (switches)
+    (dolist (s (string-split dired-actual-switches))
+      (when (string-match "\\`-[^-]" s)
+        (setq s (remove ?b s)))
+      (unless (or (string= s "-")
+                  (string-match "escape" s))
+        (cl-pushnew s switches :test 'equal)))
+    (mapconcat #'identity (nreverse switches) " ")))
+
+(defun dired--toggle-b-switch ()
+  "Add or remove `b' switch and redisplay Dired buffer.
+When the current Dired buffer has a file name containing a newline, add
+the `b' switch to the actual switches if it isn't already among them;
+otherwise remove the `b' switch unless it is in `dired-listing-switches'.
+Then redisplay the Dired buffer.  This function is called from
+`post-command-hook' in Dired mode buffers."
+  (when (eq major-mode 'dired-mode)
+    (if (and (dired--filename-with-newline-p) dired-auto-toggle-b-switch)
+        (unless (dired-switches-escape-p dired-actual-switches)
+          (setq dired-actual-switches (concat dired-actual-switches " -b"))
+          (dired-revert))
+      (unless (dired-switches-escape-p dired-listing-switches)
+        (when (dired-switches-escape-p dired-actual-switches)
+          (setq dired-actual-switches (dired--remove-b-switch))
+          (dired-revert))))))
+
+(defun dired--set-auto-toggle-b-switch (symbol value)
+  "The :set function for user option `dired-auto-toggle-b-switch'."
+  (custom-set-default symbol value)
+  (if value
+      (add-hook 'post-command-hook #'dired--toggle-b-switch nil t)
+    (remove-hook 'post-command-hook #'dired--toggle-b-switch t))
+  (dolist (b (buffer-list))
+    (with-current-buffer b
+      (dired--toggle-b-switch))))
+
+(defun dired--display-filename-with-newline-warning (dir)
+  "Display a warning if buffer DIR has a file name with a newline."
+  (let ((msg "Literal newline in file name.
+This Dired buffer displays a file name containing a literal newline character.
+Executing Dired operations on files displayed this way may fail and signal an
+error.  To avoid this you can temporarily change the display for all Dired
+buffers, so that newlines in file names appear as \"\\n\", by typing `M-:' and
+entering `(setopt dired-auto-toggle-b-switch t)' in the minibuffer.  To change
+the display only for this Dired buffer click or press RETURN `%s'.
+See `%s' for other alternatives and more information."))
+    (display-warning
+     'dired
+     (format-message
+      msg
+      (buttonize "here"
+                 (lambda (_)
+                   (pop-to-buffer dir)
+                   (when (dired--filename-with-newline-p)
+                     (unless (dired-switches-escape-p dired-actual-switches)
+                       (setq dired-actual-switches
+                             (concat dired-actual-switches " -b"))
+                       (dired-revert))))
+                 nil "mouse-2: Change newline display")
+      (buttonize "(emacs) Dired Enter"
+                 (lambda (_)
+                   (info "(emacs) Dired Enter")
+                   (declare-function Info-goto-node "info")
+                   (with-current-buffer "*info*"
+                     (Info-goto-node "File names with newline")))
+                 nil "mouse-2: Jump to Info node")))
+    ;; Display *Warnings* buffer with point at start of message instead
+    ;; of at the end.
+    (with-current-buffer "*Warnings*"
+      (set-window-point (get-buffer-window)
+                        (search-backward "Warning (dired)")))))
+
 
 ;;; Deleting files