mirrors/emacs
1
Fork 0
mirror of git://git.sv.gnu.org/emacs.git synced 2025-12-06 06:20:55 -08:00
Code Issues Projects Releases Wiki Activity

New gnus-search library

Browse source 
This library provides a fundamental reworking of the search
functionality previously found in nnir.el.  It uses class-based search
engines to interface with external searching facilities, and a parsed
search query syntax that can search multiple engines.

* lisp/gnus/gnus-search.el: New library containing search
functionality for Gnus.
* doc/misc/gnus.texi: Document.
* lisp/gnus/gnus-group.el (gnus-group-make-search-group,
gnus-group-read-ephemeral-search-group): Remove references to nnir,
change meaning of prefix argument, change values of nnselect-function
and nnselect-args.
* lisp/gnus/nnselect.el: Replace references to nnir
(nnselect-request-article): Use gnus-search functions, and search
criteria.
(nnselect-request-thread, nnselect-search-thread): Use gnus-search
thread search.
(gnus-summary-make-search-group): Switch to use gnus-search function
and arguments.
* test/lisp/gnus/search-tests.el: Tests for new functionality.
This commit is contained in:
Eric Abrahamsen 2020-10-14 21:39:46 -07:00
parent 9aa6b5bb89
commit 7fad12c59b
6 changed files with 2675 additions and 411 deletions
Download patch file Download diff file Expand all files Collapse all files

592
doc/misc/gnus.texi
View file

@ -795,19 +795,11 @@ Advanced Scoring
Searching Searching
* nnir:: Searching with various engines. * Search Engines:: Selecting and configuring search engines.
* Creating Search Groups:: Creating search groups.
* Search Queries:: Gnus' built-in search syntax.
* nnmairix:: Searching with Mairix. * nnmairix:: Searching with Mairix.
nnir
* What is nnir?:: What does nnir do.
* Basic Usage:: How to perform simple searches.
* Setting up nnir:: How to set up nnir.
Setting up nnir
* Associating Engines:: How to associate engines.
Various Various
* Process/Prefix:: A convention used by many treatment commands. * Process/Prefix:: A convention used by many treatment commands.
@ -17919,12 +17911,11 @@ Or we may wish to create a group from the results of a search query:
@lisp @lisp
(nnselect-specs (nnselect-specs
(nnselect-function . nnir-run-query) (nnselect-function . gnus-search-run-query)
(nnselect-args (nnselect-args
(nnir-query-spec (search-query-spec
(query . "FLAGGED") (query . "mark:flag"))
(criteria . "")) (search-group-spec
(nnir-group-spec
("nnimap:home") ("nnimap:home")
("nnimap:work")))) ("nnimap:work"))))
@end lisp @end lisp
@ -17945,9 +17936,8 @@ find all message that have been received recently from certain groups:
(days-to-time (car args))))) (days-to-time (car args)))))
(cons 'criteria ""))) (cons 'criteria "")))
(group-spec (cadr args))) (group-spec (cadr args)))
(nnir-run-query (cons 'nnir-specs (gnus-search-run-query (list (cons 'search-query-spec query-spec)
(list (cons 'nnir-query-spec query-spec) (cons 'search-group-spec group-spec))))))
(cons 'nnir-group-spec group-spec))))))
@end lisp @end lisp
Then the following @code{nnselect-specs}: Then the following @code{nnselect-specs}:
@ -17970,18 +17960,13 @@ parameter of @code{nnselect-rescan} will allow automatic refreshing.
A refresh can always be invoked manually through A refresh can always be invoked manually through
@code{gnus-group-get-new-news-this-group}. @code{gnus-group-get-new-news-this-group}.
The nnir interface (@pxref{nnir}) includes engines for searching a Gnus includes engines for searching a variety of backends. While the
variety of backends. While the details of each search engine vary, details of each search engine vary, the result of a search is always a
the result of an nnir search is always a vector of the sort used by vector of the sort used by the nnselect method, and the results of
the nnselect method, and the results of nnir queries are usually queries are usually viewed using an nnselect group. Indeed the
viewed using an nnselect group. Indeed the standard search function standard search function @code{gnus-group-read-ephemeral-search-group}
@code{gnus-group-read-ephemeral-search-group} just creates an just creates an ephemeral nnselect group with the appropriate search
ephemeral nnselect group with the appropriate nnir query as the query as the @code{nnselect-specs}.
@code{nnselect-specs}. nnir originally included both the search
engines and the glue to connect search results to gnus. Over time
this glue evolved into the nnselect method. The two had a mostly
amicable parting so that nnselect could pursue its dream of becoming a
fully functioning backend, but occasional conflicts may still linger.
@node Combined Groups @node Combined Groups
@subsection Combined Groups @subsection Combined Groups
@ -21445,9 +21430,6 @@ four days, Gnus will decay the scores four times, for instance.
@chapter Searching @chapter Searching
@cindex searching @cindex searching
FIXME: A brief comparison of nnir, nnmairix, contrib/gnus-namazu would
be nice.
Gnus has various ways of finding articles that match certain criteria Gnus has various ways of finding articles that match certain criteria
(from a particular author, on a certain subject, etc.). The simplest (from a particular author, on a certain subject, etc.). The simplest
method is to enter a group and then either "limit" the summary buffer method is to enter a group and then either "limit" the summary buffer
@ -21455,50 +21437,166 @@ to the desired articles using the limiting commands (@pxref{Limiting}),
or searching through messages in the summary buffer (@pxref{Searching or searching through messages in the summary buffer (@pxref{Searching
for Articles}). for Articles}).
Limiting commands and summary buffer searching work on subsets of the Limiting commands and summary buffer searching work on articles
articles already fetched from the servers, and these commands won't already fetched from the servers, and these commands won't query the
query the server for additional articles. While simple, these methods server for additional articles. While simple, these methods are
are therefore inadequate if the desired articles span multiple groups, therefore inadequate if the desired articles span multiple groups, or
or if the group is so large that fetching all articles is impractical. if the group is so large that fetching all articles is impractical.
Many backends (such as imap, notmuch, namazu, etc.) provide their own
facilities to search for articles directly on the server and Gnus can It's possible to search a backend more thoroughly using an associated
take advantage of these methods. This chapter describes tools for search engine. Some backends come with their own search engine: IMAP
searching groups and servers for articles matching a query. servers, for instance, do their own searching. Other backends, for
example a local @code{nnmaildir} installation, might require the user
to manually set up some sort of search indexing. Default associations
between backends and engines can be defined in
@code{gnus-search-default-engines}, and engines can also be defined on
a per-backend basis (@pxref{Search Engines}).
Once the search engines are set up, you can search for messages in
groups from one or more backends, and show the results in a group.
The groups that hold search results are created on the nnselect
backend, and can be either ephemeral or persistent (@pxref{Creating
Search Groups}).
@vindex gnus-search-use-parsed-queries
Search queries can be specified one of two ways: either using the
syntax of the engine responsible for the group you're searching, or
using Gnus' generalized search syntax. Set the option
@code{gnus-search-use-parsed-queries} to a non-nil value to used the
generalized syntax. The advantage of this syntax is that, if you have
multiple backends indexed by different engines, you don't need to
remember which one you're searching---it's also possible to issue the
same query against multiple groups, indexed by different engines, at
the same time. It also provides a few other conveniences including
relative date parsing and tie-ins into other Emacs packages. For
details on Gnus' query language, see @ref{Search Queries}.
@menu @menu
* nnir:: Searching with various engines. * Search Engines:: Selecting and configuring search engines.
* nnmairix:: Searching with Mairix. * Creating Search Groups:: How and where.
* Search Queries:: Gnus' built-in search syntax.
* nnmairix:: Searching with Mairix.
@end menu @end menu
@node nnir @node Search Engines
@section nnir @section Search Engines
@cindex nnir @cindex search engines
@cindex configuring search
This section describes how to use @code{nnir} to search for articles In order to search for messages from any given server, that server
within gnus. must have a search engine associated with it. IMAP servers do their
own searching (theoretically it is possible to use a different engine
to search an IMAP store, but we don't recommend it), but in all other
cases the user will have to manually specify an engine to use. This
can be done at two different levels: by server type, or on a
per-server basis.
@menu @vindex gnus-search-default-engines
* What is nnir?:: What does @code{nnir} do? The option @code{gnus-search-default-engines} assigns search engines
* Basic Usage:: How to perform simple searches. by server type. Its value is an alist mapping symbols indicating a
* Setting up nnir:: How to set up @code{nnir}. server type (e.g.@: @code{nnmaildir} or @code{nnml}) to symbols
@end menu indicating a search engine class. The built-in search engine symbols
are:
@node What is nnir? @itemize
@subsection What is nnir? @item
@code{gnus-search-imap}
@code{nnir} is a Gnus interface to a number of tools for searching @item
through mail and news repositories. Different backends (like @code{gnus-search-find-grep}
@code{nnimap} and @code{nntp}) work with different tools (called
@dfn{engines} in @code{nnir} lingo), but all use the same basic search
interface.
The @code{nnimap} search engine should work with no configuration. @item
Other engines may require a local index that needs to be created and @code{gnus-search-notmuch}
maintained outside of Gnus.
@item
@code{gnus-search-swish-e}
@node Basic Usage @item
@subsection Basic Usage @code{gnus-search-swish++}
@item
@code{gnus-search-mairix}
@item
@code{gnus-search-namazu}
@end itemize
If you need more granularity, you can specify a search engine in the
server definition, using the @code{gnus-search-engine} key, whether
that be in your @file{.gnus.el} config file, or through Gnus' server
buffer. That might look like:
@example
'(nnmaildir "My Mail"
(directory "/home/user/.mail")
(gnus-search-engine gnus-search-notmuch
(config-file "/home/user/.mail/.notmuch_config")))
@end example
Search engines like notmuch, namazu and mairix are similar in
behavior: they use a local executable to create an index of a message
store, and run command line search queries against those messages,
and return a list of absolute file names of matching messages.
These engines have a handful of configuration parameters in common.
These common parameters are:
@table @code
@item program
The name of the executable. Defaults to the plain
program name such as @command{notmuch} or @command{namazu}.
@item config-file
The absolute filename of the configuration file for this search
engine.
@item remove-prefix
The directory part to be removed from the filenames returned by the
search query. This absolute path should include everything up to the
top level of the message store.
@item switches
Additional command-line switches to be fed to the search program. The
value of this parameter must be a list of strings, one string per
switch.
@end table
The options above can be set in one of two ways: using a customization
option that is set for all engines of that type, or on a per-engine
basis in your server configuration files.
The customization options are formed on the pattern
@code{gnus-search-@var{engine}-@var{parameter}}. For instance, to use a
non-standard notmuch program, you might set
@code{gnus-search-notmuch-program} to @file{/usr/local/bin/notmuch}.
This would apply to all notmuch engines. The engines that use these
options are: ``notmuch'', ``namazu'', ``mairix'', ``swish-e'' and
``swish++''.
Alternately, the options can be set directly on your Gnus server
definitions, for instance, in the @code{nnmaildir} example above.
Note that the server options are part of the @code{gnus-search-engine}
sexp, and the option symbol and value form a two-element list, not a
cons cell.
The namazu and swish-e engines each have one additional option,
specifying where to store their index files. For namazu it is
@code{index-directory}, and should be a single directory path. For
swish-e it is @code{index-files}, and should be a list of strings.
All indexed search engines come with their own method of updating
their search indexes to include newly-arrived messages. Gnus
currently provides no convenient interface for this, and you'll have
to manage updates yourself, though this will likely change in the
future.
Lastly, all search engines accept a @code{raw-queries-p} option. This
indicates that engines of this type (or this particular engine) should
always use raw queries, never parsed (@pxref{Search Queries}).
@node Creating Search Groups
@section Creating Search Groups
@cindex creating search groups
In the group buffer typing @kbd{G G} will search the group on the In the group buffer typing @kbd{G G} will search the group on the
current line by calling @code{gnus-group-read-ephemeral-search-group}. current line by calling @code{gnus-group-read-ephemeral-search-group}.
@ -21525,297 +21623,133 @@ in their original group. You can @emph{warp} (i.e., jump) to the
original group for the article on the current line with @kbd{A W}, aka original group for the article on the current line with @kbd{A W}, aka
@code{gnus-warp-to-article}. @code{gnus-warp-to-article}.
You say you want to search more than just the group on the current line? You say you want to search more than just the group on the current
No problem: just process-mark the groups you want to search. You want line? No problem: just process-mark the groups you want to search.
even more? Calling for an nnir search with the cursor on a topic heading You want even more? Initiating a search with the cursor on a topic
will search all the groups under that heading. heading will search all the groups under that topic.
@vindex gnus-search-ignored-newsgroups
Still not enough? OK, in the server buffer Still not enough? OK, in the server buffer
@code{gnus-group-read-ephemeral-search-group} (now bound to @kbd{G}) @code{gnus-group-read-ephemeral-search-group} (here bound to @kbd{G})
will search all groups from the server on the current line. Too much? will search all groups from the server on the current line. Too much?
Want to ignore certain groups when searching, like spam groups? Just Want to ignore certain groups when searching, like spam groups? Just
customize @code{nnir-ignored-newsgroups}. customize @code{gnus-search-ignored-newsgroups}: groups matching this
regexp will not be searched.
One more thing: individual search engines may have special search @node Search Queries
features. You can access these special features by giving a @section Search Queries
prefix-arg to @code{gnus-group-read-ephemeral-search-group}. If you @cindex search queries
are searching multiple groups with different search engines you will @cindex search syntax
be prompted for the special search features for each engine
separately.
Gnus provides an optional unified search syntax that can be used
across all supported search engines. This can be convenient in that
you don't have to remember different search syntaxes; it's also
possible to mark multiple groups indexed by different engines and
issue a single search against them.
@node Setting up nnir @vindex gnus-search-use-parsed-queries
@subsection Setting up nnir Set the option @code{gnus-search-use-parsed-queries} to non-@code{nil}
to enable this---it is @code{nil} by default. Even if it is
non-@code{nil}, it's still possible to turn off parsing for a class of
engines or a single engine (@pxref{Search Engines}), or a single
search by giving a prefix argument to any of the search commands.
To set up nnir you may need to do some prep work. Firstly, you may The search syntax is fairly simple: keys and values are separated by a
need to configure the search engines you plan to use. Some of them, colon, multi-word values must be quoted, ``and'' is implicit, ``or''
like @code{imap}, need no special configuration. Others, like is explicit, ``not'' will negate the following expression (or keys can
@code{namazu} and @code{swish}, require configuration as described be prefixed with a ``-''),and parentheses can be used to group logical
below. Secondly, you need to associate a search engine with a server sub-clauses. For example:
or a backend.
If you just want to use the @code{imap} engine to search @code{nnimap} @example
servers then you don't have to do anything. But you might want to (from:john or from:peter) subject:"lunch tomorrow" since:3d
read the details of the query language anyway. @end example
@menu The syntax is made to be accepted by a wide range of engines, and thus
* Associating Engines:: How to associate engines. will happily accept most input, valid or not. Some terms will only be
* The imap Engine:: Imap configuration and usage. meaningful to some engines; other engines will drop them silently.
* The swish++ Engine:: Swish++ configuration and usage.
* The swish-e Engine:: Swish-e configuration and usage.
* The namazu Engine:: Namazu configuration and usage.
* The notmuch Engine:: Notmuch configuration and usage.
* The hyrex Engine:: Hyrex configuration and usage.
* Customizations:: User customizable settings.
@end menu
@node Associating Engines Key completion is offered on @key{TAB}, but it's also possible to
@subsubsection Associating Engines enter the query with abbreviated keys, which will be expanded during
parsing. If a key is abbreviated to the point of ambiguity (for
instance, ``s:'' could be ``subject:'' or ``since:''), an error will
be raised.
Supported keys include all the usual mail headers: ``from'',
When searching a group, @code{nnir} needs to know which search engine to ``subject'', ``cc'', etc. Other keys are:
use. You can configure a given server to use a particular engine by
setting the server variable @code{nnir-search-engine} to the engine
name. For example to use the @code{namazu} engine to search the server
named @code{home} you can use
@lisp
(setq gnus-secondary-select-methods
'((nnml "home"
(nnimap-address "localhost")
(nnir-search-engine namazu))))
@end lisp
Alternatively you might want to use a particular engine for all servers
with a given backend. For example, you might want to use the @code{imap}
engine for all servers using the @code{nnimap} backend. In this case you
can customize the variable @code{nnir-method-default-engines}. This is
an alist of pairs of the form @code{(backend . engine)}. By default this
variable is set to use the @code{imap} engine for all servers using the
@code{nnimap} backend. But if you wanted to use @code{namazu} for all
your servers with an @code{nnimap} backend you could change this to
@lisp
'((nnimap . namazu))
@end lisp
@node The imap Engine
@subsubsection The imap Engine
The @code{imap} engine requires no configuration.
Queries using the @code{imap} engine follow a simple query language.
The search is always case-insensitive and supports the following
features (inspired by the Google search input language):
@table @samp @table @samp
@item body
@item Boolean query operators The body of the message.
AND, OR, and NOT are supported, and parentheses can be used to control @item recipient
operator precedence, e.g., (emacs OR xemacs) AND linux. Note that Equivalent to @samp{to or cc or bcc}.
operators must be written with all capital letters to be @item address
recognized. Also preceding a term with a @minus{} sign is equivalent Equivalent to @samp{from or recipient}.
to NOT term. @item id
The keys @samp{message-id} and @samp{id} are equivalent.
@item Automatic AND queries @item mark
If you specify multiple words then they will be treated as an AND Accepts @samp{flag}, @samp{seen}, @samp{read} or @samp{replied}, or
expression intended to match all components. any of Gnus' single-letter representations of those marks, e.g.@:
@samp{mark:R} for @samp{read}.
@item Phrase searches @item tag
If you wrap your query in double-quotes then it will be treated as a This is interpreted as @samp{keyword} for IMAP and @samp{tag} for
literal string. notmuch.
@item attachment
Matches the attachment file name.
@item before
Date is exclusive; see below for date parsing.
@item after
Date is inclusive; can also use @samp{since}.
@item thread
Return entire message threads, not just individual messages.
@item raw
Do not parse this particular search.
@item limit
Limit the results to this many messages. When searching multiple
groups this may give undesired results, as the limiting happens before
sorting.
@item grep
Only applicable to ``local index'' engines such as mairix or notmuch.
On systems with a grep command, additionally filter the results by
using the value of this term as a grep regexp.
@end table @end table
By default the whole message will be searched. The query can be limited @vindex gnus-search-contact-tables
to a specific part of a message by using a prefix-arg. After inputting Elisp-based contact management packages (e.g.@: BBDB or EBDB) can push
the query this will prompt (with completion) for a message part. completion tables onto the variable @code{gnus-search-contact-tables},
Choices include ``Whole message'', ``Subject'', ``From'', and allowing auto-completion of contact names and addresses for keys like
``To''. Any unrecognized input is interpreted as a header name. For @samp{from} or @samp{to}.
example, typing @kbd{Message-ID} in response to this prompt will limit
the query to the Message-ID header.
Finally selecting ``Imap'' will interpret the query as a raw @subsection Date value parsing
@acronym{IMAP} search query. The format of such queries can be found in
RFC3501.
If you don't like the default of searching whole messages you can @vindex gnus-search-date-keys
customize @code{nnir-imap-default-search-key}. For example to use Date-type keys (see @code{gnus-search-date-keys}) will accept a wide
@acronym{IMAP} queries by default variety of values. First, anything that @code{parse-time-string} can
parse is acceptable. Dates with missing values will be interpreted as
the most recent occurrence thereof: for instance ``march 03'' is the
most recent March 3rd. Lastly, it's possible to use relative
specifications, such as ``3d'' (three days ago). This format also accepts
w, m and y.
@lisp When creating persistent search groups, the search is saved unparsed,
(setq nnir-imap-default-search-key "Imap") and re-parsed every time the group is updated. So a permanent search
@end lisp group with a query like:
@node The swish++ Engine
@subsubsection The swish++ Engine
FIXME: Say something more here.
Documentation for swish++ may be found at the swish++ sourceforge page:
@uref{http://swishplusplus.sourceforge.net}
@table @code
@item nnir-swish++-program
The name of the swish++ executable. Defaults to @code{search}
@item nnir-swish++-additional-switches
A list of strings to be given as additional arguments to
swish++. @code{nil} by default.
@item nnir-swish++-remove-prefix
The prefix to remove from each file name returned by swish++ in order
to get a group name. By default this is @code{$HOME/Mail}.
@end table
@node The swish-e Engine
@subsubsection The swish-e Engine
FIXME: Say something more here.
@table @code
@item nnir-swish-e-program
The name of the swish-e search program. Defaults to @code{swish-e}.
@item nnir-swish-e-additional-switches
A list of strings to be given as additional arguments to
swish-e. @code{nil} by default.
@item nnir-swish-e-remove-prefix
The prefix to remove from each file name returned by swish-e in order
to get a group name. By default this is @code{$HOME/Mail}.
@end table
@node The namazu Engine
@subsubsection The namazu Engine
Using the namazu engine requires creating and maintaining index files.
One directory should contain all the index files, and nnir must be told
where to find them by setting the @code{nnir-namazu-index-directory}
variable.
To work correctly the @code{nnir-namazu-remove-prefix} variable must
also be correct. This is the prefix to remove from each file name
returned by Namazu in order to get a proper group name (albeit with @samp{/}
instead of @samp{.}).
For example, suppose that Namazu returns file names such as
@samp{/home/john/Mail/mail/misc/42}. For this example, use the
following setting: @code{(setq nnir-namazu-remove-prefix
"/home/john/Mail/")} Note the trailing slash. Removing this prefix from
the directory gives @samp{mail/misc/42}. @code{nnir} knows to remove
the @samp{/42} and to replace @samp{/} with @samp{.} to arrive at the
correct group name @samp{mail.misc}.
Extra switches may be passed to the namazu search command by setting the
variable @code{nnir-namazu-additional-switches}. It is particularly
important not to pass any switches to namazu that will change the
output format. Good switches to use include @option{--sort},
@option{--ascending}, @option{--early} and @option{--late}.
Refer to the Namazu documentation for further
information on valid switches.
Mail must first be indexed with the @command{mknmz} program. Read the
documentation for namazu to create a configuration file. Here is an
example:
@cartouche
@example
package conf; # Don't remove this line!
# Paths which will not be indexed. Don't use '^' or '$' anchors.
$EXCLUDE_PATH = "spam|sent";
# Header fields which should be searchable. case-insensitive
$REMAIN_HEADER = "from|date|message-id|subject";
# Searchable fields. case-insensitive
$SEARCH_FIELD = "from|date|message-id|subject";
# The max length of a word.
$WORD_LENG_MAX = 128;
# The max length of a field.
$MAX_FIELD_LENGTH = 256;
@end example
@end cartouche
For this example, mail is stored in the directories @samp{~/Mail/mail/},
@samp{~/Mail/lists/} and @samp{~/Mail/archive/}, so to index them go to
the index directory set in @code{nnir-namazu-index-directory} and issue
the following command:
@example @example
mknmz --mailnews ~/Mail/archive/ ~/Mail/mail/ ~/Mail/lists/ from:"my boss" mark:flag since:1w
@end example @end example
For maximum searching efficiency you might want to have a cron job run would always contain only messages from the past seven days.
this command periodically, say every four hours.
@node The notmuch Engine
@subsubsection The notmuch Engine
@table @code
@item nnir-notmuch-program
The name of the notmuch search executable. Defaults to
@samp{notmuch}.
@item nnir-notmuch-additional-switches
A list of strings, to be given as additional arguments to notmuch.
@item nnir-notmuch-remove-prefix
The prefix to remove from each file name returned by notmuch in order
to get a group name (albeit with @samp{/} instead of @samp{.}). This
is a regular expression.
@item nnir-notmuch-filter-group-names-function
A function used to transform the names of groups being searched in,
for use as a ``path:'' search keyword for notmuch. If nil, the
default, ``path:'' keywords are not used. Otherwise, this should be a
callable which accepts a single group name and returns a transformed
name as notmuch expects to see it. In many mail backends, for
instance, dots in group names must be converted to forward slashes: to
achieve this, set this option to
@example
(lambda (g) (replace-regexp-in-string "\\." "/" g))
@end example
@end table
@node The hyrex Engine
@subsubsection The hyrex Engine
This engine is obsolete.
@node Customizations
@subsubsection Customizations
@table @code
@item nnir-method-default-engines
Alist of pairs of server backends and search engines. The default
association is
@example
(nnimap . imap)
@end example
@item nnir-ignored-newsgroups
A regexp to match newsgroups in the active file that should be skipped
when searching all groups on a server.
@end table
@node nnmairix @node nnmairix
@section nnmairix @section nnmairix
@cindex mairix @cindex mairix
@cindex nnmairix @cindex nnmairix
This section is now mostly obsolete, as mairix can be used as a regular
search engine, including persistent search groups, with
@code{nnselect}.
This paragraph describes how to set up mairix and the back end This paragraph describes how to set up mairix and the back end
@code{nnmairix} for indexing and searching your mail from within @code{nnmairix} for indexing and searching your mail from within
Gnus. Additionally, you can create permanent ``smart'' groups which are Gnus. Additionally, you can create permanent ``smart'' groups which are

8
etc/NEWS
View file

@ -454,7 +454,13 @@ tags to be considered as well.
** Gnus ** Gnus
+++ +++
*** New value for user option 'smiley-style'. *** New gnus-search library
A new unified search syntax which can be used across multiple
supported search engines. Set 'gnus-search-use-parsed-queries' to
non-nil to enable.
+++
*** New value for user option 'smiley-style'
Smileys can now be rendered with emojis instead of small images when Smileys can now be rendered with emojis instead of small images when
using the new 'emoji' value in 'smiley-style'. using the new 'emoji' value in 'smiley-style'.

68
lisp/gnus/gnus-group.el
View file

@ -3165,29 +3165,27 @@ mail messages or news articles in files that have numeric names."
(gnus-group-real-name group) (gnus-group-real-name group)
(list 'nndir (gnus-group-real-name group) (list 'nndir-directory dir))))) (list 'nndir (gnus-group-real-name group) (list 'nndir-directory dir)))))
(autoload 'nnir-read-parms "nnir")
(autoload 'nnir-server-to-search-engine "nnir")
(autoload 'gnus-group-topic-name "gnus-topic") (autoload 'gnus-group-topic-name "gnus-topic")
(autoload 'gnus-search-make-spec "gnus-search")
;; Temporary to make group creation easier ;; Temporary to make group creation easier
(defun gnus-group-make-search-group (nnir-extra-parms &optional specs) (defun gnus-group-make-search-group (no-parse &optional specs)
"Make a group based on a search. "Make a group based on a search.
Prompt for a search query and determine the groups to search as Prompt for a search query and determine the groups to search as
follows: if called from the *Server* buffer search all groups follows: if called from the *Server* buffer search all groups
belonging to the server on the current line; if called from the belonging to the server on the current line; if called from the
*Group* buffer search any marked groups, or the group on the *Group* buffer search any marked groups, or the group on the
current line, or all the groups under the current topic. Calling current line, or all the groups under the current topic. A
with a prefix arg prompts for additional search-engine specific prefix arg NO-PARSE means that Gnus should not parse the search
constraints. A non-nil SPECS arg must be an alist with query before passing it to the underlying search engine. A
`nnir-query-spec' and `nnir-group-spec' keys, and skips all non-nil SPECS arg must be an alist with `search-query-spec' and
prompting." `search-group-spec' keys, and skips all prompting."
(interactive "P") (interactive "P")
(let ((name (gnus-read-group "Group name: "))) (let ((name (gnus-read-group "Group name: ")))
(with-current-buffer gnus-group-buffer (with-current-buffer gnus-group-buffer
(let* ((group-spec (let* ((group-spec
(or (or
(cdr (assq 'nnir-group-spec specs)) (cdr (assq 'search-group-spec specs))
(if (gnus-server-server-name) (if (gnus-server-server-name)
(list (list (gnus-server-server-name))) (list (list (gnus-server-server-name)))
(seq-group-by (seq-group-by
@ -3199,16 +3197,8 @@ prompting."
(assoc (gnus-group-topic-name) gnus-topic-alist)))))))) (assoc (gnus-group-topic-name) gnus-topic-alist))))))))
(query-spec (query-spec
(or (or
(cdr (assq 'nnir-query-spec specs)) (cdr (assq 'search-query-spec specs))
(apply (gnus-search-make-spec no-parse))))
'append
(list (cons 'query
(read-string "Query: " nil 'nnir-search-history)))
(when nnir-extra-parms
(mapcar
(lambda (x)
(nnir-read-parms (nnir-server-to-search-engine (car x))))
group-spec))))))
(gnus-group-make-group (gnus-group-make-group
name name
(list 'nnselect "nnselect") (list 'nnselect "nnselect")
@ -3216,29 +3206,29 @@ prompting."
(list (list
(cons 'nnselect-specs (cons 'nnselect-specs
(list (list
(cons 'nnselect-function 'nnir-run-query) (cons 'nnselect-function 'gnus-search-run-query)
(cons 'nnselect-args (cons 'nnselect-args
(list (cons 'nnir-query-spec query-spec) (list (cons 'search-query-spec query-spec)
(cons 'nnir-group-spec group-spec))))) (cons 'search-group-spec group-spec)))))
(cons 'nnselect-artlist nil))))))) (cons 'nnselect-artlist nil)))))))
(define-obsolete-function-alias 'gnus-group-make-nnir-group (define-obsolete-function-alias 'gnus-group-make-nnir-group
'gnus-group-read-ephemeral-search-group "28.1") 'gnus-group-read-ephemeral-search-group "28.1")
(defun gnus-group-read-ephemeral-search-group (nnir-extra-parms &optional specs) (defun gnus-group-read-ephemeral-search-group (no-parse &optional specs)
"Read an nnselect group based on a search. "Read an nnselect group based on a search.
Prompt for a search query and determine the groups to search as Prompt for a search query and determine the groups to search as
follows: if called from the *Server* buffer search all groups follows: if called from the *Server* buffer search all groups
belonging to the server on the current line; if called from the belonging to the server on the current line; if called from the
*Group* buffer search any marked groups, or the group on the *Group* buffer search any marked groups, or the group on the
current line, or all the groups under the current topic. Calling current line, or all the groups under the current topic. A
with a prefix arg prompts for additional search-engine specific prefix arg NO-PARSE means that Gnus should not parse the search
constraints. A non-nil SPECS arg must be an alist with query before passing it to the underlying search engine. A
`nnir-query-spec' and `nnir-group-spec' keys, and skips all non-nil SPECS arg must be an alist with `search-query-spec' and
prompting." `search-group-spec' keys, and skips all prompting."
(interactive "P") (interactive "P")
(let* ((group-spec (let* ((group-spec
(or (cdr (assq 'nnir-group-spec specs)) (or (cdr (assq 'search-group-spec specs))
(if (gnus-server-server-name) (if (gnus-server-server-name)
(list (list (gnus-server-server-name))) (list (list (gnus-server-server-name)))
(seq-group-by (seq-group-by
@ -3249,16 +3239,8 @@ prompting."
(cdr (cdr
(assoc (gnus-group-topic-name) gnus-topic-alist)))))))) (assoc (gnus-group-topic-name) gnus-topic-alist))))))))
(query-spec (query-spec
(or (cdr (assq 'nnir-query-spec specs)) (or (cdr (assq 'search-query-spec specs))
(apply (gnus-search-make-spec no-parse))))
'append
(list (cons 'query
(read-string "Query: " nil 'nnir-search-history)))
(when nnir-extra-parms
(mapcar
(lambda (x)
(nnir-read-parms (nnir-server-to-search-engine (car x))))
group-spec))))))
(gnus-group-read-ephemeral-group (gnus-group-read-ephemeral-group
(concat "nnselect-" (message-unique-id)) (concat "nnselect-" (message-unique-id))
(list 'nnselect "nnselect") (list 'nnselect "nnselect")
@ -3268,10 +3250,10 @@ prompting."
(list (list
(cons 'nnselect-specs (cons 'nnselect-specs
(list (list
(cons 'nnselect-function 'nnir-run-query) (cons 'nnselect-function 'gnus-search-run-query)
(cons 'nnselect-args (cons 'nnselect-args
(list (cons 'nnir-query-spec query-spec) (list (cons 'search-query-spec query-spec)
(cons 'nnir-group-spec group-spec))))) (cons 'search-group-spec group-spec)))))
(cons 'nnselect-artlist nil))))) (cons 'nnselect-artlist nil)))))
(defun gnus-group-add-to-virtual (n vgroup) (defun gnus-group-add-to-virtual (n vgroup)

2231
lisp/gnus/gnus-search.el Normal file
View file

File diff suppressed because it is too large Load diff

91
lisp/gnus/nnselect.el
View file

@ -36,10 +36,10 @@
;; sorting. Most functions will just chose a fixed number, such as ;; sorting. Most functions will just chose a fixed number, such as
;; 100, for this score. ;; 100, for this score.
;; For example the search function `nnir-run-query' applied to ;; For example the search function `gnus-search-run-query' applied to
;; arguments specifying a search query (see "nnir.el") can be used to ;; arguments specifying a search query (see "gnus-search.el") can be
;; return a list of articles from a search. Or the function can be the ;; used to return a list of articles from a search. Or the function
;; identity and the args a vector of articles. ;; can be the identity and the args a vector of articles.
;;; Code: ;;; Code:
@ -47,7 +47,7 @@
;;; Setup: ;;; Setup:
(require 'gnus-art) (require 'gnus-art)
(require 'nnir) (require 'gnus-search)
(eval-when-compile (require 'cl-lib)) (eval-when-compile (require 'cl-lib))
@ -372,25 +372,25 @@ If this variable is nil, or if the provided function returns nil,
;; find the servers for a pseudo-article ;; find the servers for a pseudo-article
(if (eq 'nnselect (car (gnus-server-to-method server))) (if (eq 'nnselect (car (gnus-server-to-method server)))
(with-current-buffer gnus-summary-buffer (with-current-buffer gnus-summary-buffer
(let ((thread (gnus-id-to-thread article))) (let ((thread (gnus-id-to-thread article)))
(when thread (when thread
(mapc (mapc
#'(lambda (x) (lambda (x)
(when (and x (> x 0)) (when (and x (> x 0))
(cl-pushnew (cl-pushnew
(list (list
(gnus-method-to-server (gnus-method-to-server
(gnus-find-method-for-group (gnus-find-method-for-group
(nnselect-article-group x)))) servers :test 'equal))) (nnselect-article-group x)))) servers :test 'equal)))
(gnus-articles-in-thread thread))))) (gnus-articles-in-thread thread)))))
(setq servers (list (list server)))) (setq servers (list (list server))))
(setq artlist (setq artlist
(nnir-run-query (gnus-search-run-query
(list (list
(cons 'nnir-query-spec (cons 'search-query-spec
(list (cons 'query (format "HEADER Message-ID %s" article)) (list (cons 'query `((id . ,article)))
(cons 'criteria "") (cons 'shortcut t))) (cons 'criteria "") (cons 'shortcut t)))
(cons 'nnir-group-spec servers)))) (cons 'search-group-spec servers))))
(unless (zerop (nnselect-artlist-length artlist)) (unless (zerop (nnselect-artlist-length artlist))
(setq (setq
group-art group-art
@ -603,26 +603,35 @@ If this variable is nil, or if the provided function returns nil,
(cl-some #'(lambda (x) (cl-some #'(lambda (x)
(when (and x (> x 0)) x)) (when (and x (> x 0)) x))
(gnus-articles-in-thread thread))))))))) (gnus-articles-in-thread thread)))))))))
;; Check if we are dealing with an imap backend. ;; Check if search-based thread referral is permitted, and
(if (eq 'nnimap ;; available.
(car (gnus-find-method-for-group artgroup))) (if (and gnus-refer-thread-use-search
(gnus-search-server-to-engine
(gnus-method-to-server
(gnus-find-method-for-group artgroup))))
;; If so we perform the query, massage the result, and return ;; If so we perform the query, massage the result, and return
;; the new headers back to the caller to incorporate into the ;; the new headers back to the caller to incorporate into the
;; current summary buffer. ;; current summary buffer.
(let* ((group-spec (let* ((group-spec
(list (delq nil (list (list (delq nil (list
(or server (gnus-group-server artgroup)) (or server (gnus-group-server artgroup))
(unless gnus-refer-thread-use-search (unless gnus-refer-thread-use-search
artgroup))))) artgroup)))))
(ids (cons (mail-header-id header)
(split-string
(or (mail-header-references header)
""))))
(query-spec (query-spec
(list (cons 'query (nnimap-make-thread-query header)) (list (cons 'query (mapconcat (lambda (i)
(cons 'criteria ""))) (format "id:%s" i))
ids " or "))
(cons 'thread t)))
(last (nnselect-artlist-length gnus-newsgroup-selection)) (last (nnselect-artlist-length gnus-newsgroup-selection))
(first (1+ last)) (first (1+ last))
(new-nnselect-artlist (new-nnselect-artlist
(nnir-run-query (gnus-search-run-query
(list (cons 'nnir-query-spec query-spec) (list (cons 'search-query-spec query-spec)
(cons 'nnir-group-spec group-spec)))) (cons 'search-group-spec group-spec))))
old-arts seq old-arts seq
headers) headers)
(mapc (mapc
@ -670,7 +679,7 @@ If this variable is nil, or if the provided function returns nil,
group group
(cons 1 (nnselect-artlist-length gnus-newsgroup-selection)))) (cons 1 (nnselect-artlist-length gnus-newsgroup-selection))))
headers) headers)
;; If not an imap backend just warp to the original article ;; If we can't or won't use search, just warp to the original
;; group and punt back to gnus-summary-refer-thread. ;; group and punt back to gnus-summary-refer-thread.
(and (gnus-warp-to-article) (gnus-summary-refer-thread)))))) (and (gnus-warp-to-article) (gnus-summary-refer-thread))))))
@ -768,9 +777,15 @@ Return an article list."
The current server will be searched. If the registry is The current server will be searched. If the registry is
installed, the server that the registry reports the current installed, the server that the registry reports the current
article came from is also searched." article came from is also searched."
(let* ((query (let* ((ids (cons (mail-header-id header)
(list (cons 'query (nnimap-make-thread-query header)) (split-string
(cons 'criteria ""))) (or (mail-header-references header)
""))))
(query
(list (cons 'query (mapconcat (lambda (i)
(format "id:%s" i))
ids " or "))
(cons 'thread t)))
(server (server
(list (list (gnus-method-to-server (list (list (gnus-method-to-server
(gnus-find-method-for-group gnus-newsgroup-name))))) (gnus-find-method-for-group gnus-newsgroup-name)))))
@ -794,10 +809,10 @@ article came from is also searched."
(list (list
(cons 'nnselect-specs (cons 'nnselect-specs
(list (list
(cons 'nnselect-function 'nnir-run-query) (cons 'nnselect-function 'gnus-search-run-query)
(cons 'nnselect-args (cons 'nnselect-args
(list (cons 'nnir-query-spec query) (list (cons 'search-query-spec query)
(cons 'nnir-group-spec server))))) (cons 'search-group-spec server)))))
(cons 'nnselect-artlist nil))) (cons 'nnselect-artlist nil)))
(gnus-summary-goto-subject (gnus-id-to-article (mail-header-id header))))) (gnus-summary-goto-subject (gnus-id-to-article (mail-header-id header)))))
@ -929,18 +944,18 @@ article came from is also searched."
(declare-function gnus-registry-get-id-key "gnus-registry" (id key)) (declare-function gnus-registry-get-id-key "gnus-registry" (id key))
(defun gnus-summary-make-search-group (nnir-extra-parms) (defun gnus-summary-make-search-group (no-parse)
"Search a group from the summary buffer. "Search a group from the summary buffer.
Pass NNIR-EXTRA-PARMS on to the search engine." Pass NO-PARSE on to the search engine."
(interactive "P") (interactive "P")
(gnus-warp-to-article) (gnus-warp-to-article)
(let ((spec (let ((spec
(list (list
(cons 'nnir-group-spec (cons 'search-group-spec
(list (list (list (list
(gnus-group-server gnus-newsgroup-name) (gnus-group-server gnus-newsgroup-name)
gnus-newsgroup-name)))))) gnus-newsgroup-name))))))
(gnus-group-make-search-group nnir-extra-parms spec))) (gnus-group-make-search-group no-parse spec)))
;; The end. ;; The end.

96
test/lisp/gnus/gnus-search-tests.el Normal file
View file

@ -0,0 +1,96 @@
;;; gnus-search-tests.el --- Tests for Gnus' search routines -*- lexical-binding: t; -*-
;; Copyright (C) 2017 Free Software Foundation, Inc.
;; Author: Eric Abrahamsen <eric@ericabrahamsen.net>
;; Keywords:
;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.
;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see <http://www.gnu.org/licenses/>.
;;; Commentary:
;; Tests for the search parsing, search engines, and their
;; transformations.
;;; Code:
(require 'ert)
(require 'gnus-search)
(ert-deftest gnus-s-parse ()
"Test basic structural parsing."
(let ((pairs
'(("string" . ("string"))
("from:john" . ((from . "john")))
("here and there" . ("here" and "there"))
("here or there" . ((or "here" "there")))
("here (there or elsewhere)" . ("here" ((or "there" "elsewhere"))))
("here not there" . ("here" (not "there")))
("from:boss or not vacation" . ((or (from . "boss") (not "vacation")))))))
(dolist (p pairs)
(should (equal (gnus-search-parse-query (car p)) (cdr p))))))
(ert-deftest gnus-s-expand-keyword ()
"Test expansion of keywords"
(let ((gnus-search-expandable-keys
(default-value 'gnus-search-expandable-keys))
(pairs
'(("su" . "subject")
("sin" . "since"))))
(dolist (p pairs)
(should (equal (gnus-search-query-expand-key (car p))
(cdr p))))
(should-error (gnus-search-query-expand-key "s")
:type 'gnus-search-parse-error)))
(ert-deftest gnus-s-parse-date ()
"Test parsing of date expressions."
(let ((rel-date (encode-time 0 0 0 15 4 2017))
(pairs
'(("January" . (nil 1 nil))
("2017" . (nil nil 2017))
("15" . (15 nil nil))
("January 15" . (15 1 nil))
("tuesday" . (11 4 2017))
("1d" . (14 4 2017))
("1w" . (8 4 2017)))))
(dolist (p pairs)
(should (equal (gnus-search-query-parse-date (car p) rel-date)
(cdr p))))))
(ert-deftest gnus-s-delimited-string ()
"Test proper functioning of `gnus-search-query-return-string'."
(with-temp-buffer
(insert "one\ntwo words\nthree \"words with quotes\"\n\"quotes at start\"\n/alternate \"quotes\"/\n(more bits)")
(goto-char (point-min))
(should (string= (gnus-search-query-return-string)
"one"))
(forward-line)
(should (string= (gnus-search-query-return-string)
"two"))
(forward-line)
(should (string= (gnus-search-query-return-string)
"three"))
(forward-line)
(should (string= (gnus-search-query-return-string "\"")
"\"quotes at start\""))
(forward-line)
(should (string= (gnus-search-query-return-string "/")
"/alternate \"quotes\"/"))
(forward-line)
(should (string= (gnus-search-query-return-string ")" t)
"more bits"))))
(provide 'gnus-search-tests)
;;; search-tests.el ends here