NAME
INSTALL - Swish-e Installation Instructions
OVERVIEW
This document describes how to download, build, and install Swish-e from
source. Also found below is a basic overview of using Swish-e to index
documents, with pointers to other, more advanced examples.
This document also provides instructions on how to get help installing
and using Swish-e (and the important information you should provide when
asking for help). Please read these instructions before requesting help
on the Swish-e discussion list. See "QUESTIONS AND TROUBLESHOOTING".
Although building from source is recommended, some OS distributions
(e.g., Debian) provide pre-compiled binaries. Check with your
distribution for available packages. Build from source, if your
distribution does not offer the current version of Swish-e.
Also, please read the Swish-e FAQ (SWISH-FAQ), as it answers many
frequently-asked questions.
Swish-e knows how to index HTML, XML, and plain text documents. Helper
applications and other tools are used to convert documents such as PDF
or MS Word into a format that Swish-e can index. These additional
applications and tools (listed below) must be installed separately. The
process of converting documents is called "filtering".
NOTE: Swish-e version 4.2.0 installs a lot more files when running "make
install". Be aware that the Swish-e documentation may thus include
errors about where files are located. Please notify the Swish-e
discussion list of any documentation errors.
Upgrading from previous versions of Swish-e
If you are upgrading from a previous version of Swish-e, read the
CHANGES page first. The Swish-e index format may have changed and
existing indexes may not work with the newer version of Swish-e.
If you have existing indexes, you may need to re-index your data before
running the "make install" step described below. Swish-e may be run from
the build directory after compiling, but before installation.
Windows Users
A Windows binary version is available as a separate download from the
Swish-e site (http://swish-e.org). Many of the installation instructions
below will not apply to Windows users; the Windows version is
pre-compiled and includes libxml2, zlib, xpdf, and catdoc.
A number of Perl modules may also be needed. These can be installed with
ActiveState's PPM utility.
libwww-perl - the LWP modules (for spidering)
HTML-Tagset - used by web spider
HTML-Parser - used by web spider
MIME-Types - used for filtering documents when not spidering
HTML-Template - formatting output from swish.cgi (optional)
HTML-FillInForm (if HTML-Template is used)
Building from CVS
Please refer to the README.cvs file found in the documentation directory
$prefix/share/doc/swish-e.
SYSTEM REQUIREMENTS
Swish-e makes use of a number of libraries and tools that are not
distributed with Swish-e. Some libraries need to be installed before
building Swish-e from source; other tools can be installed at any time.
See below for details.
Software Requirements
Swish-e is written in C. It has been tested on a number of platforms,
including Sun/Solaris, Dec Alpha, BSD, Linux, Mac OS X, and Open VMS.
The GNU C compiler (gcc) and GNU make are strongly recommended. Repeat:
you will find life easier if you use the GNU tools.
Optional but Recommended Packages
Most of the packages listed below are available as easily installable
packages. Check with your operating system vendor or install them from
source. Most are very common packages that may already be installed on
your computer.
As noted below, some packages need to be installed before building
Swish-e from source, while others may be added after Swish-e is
installed.
* Libxml2
libxml2 is very strongly recommended. It is used for parsing both
HTML and XML files. Swish-e can be built and installed without
libxml2, but the HTML parser that is built into Swish-e is not as
accurate as libxml2.
http://xmlsoft.org/
libxml2 must be installed before Swish-e is built, or it will not be
used.
If libxml2 is installed in a non-standard location (e.g., libxml2 is
built with "--prefix $HOME/local"), make sure that you add the "bin"
directory to your $PATH before building Swish-e. Swish-e's configure
script uses a program created by libxml2 ("xml2-config") to find the
location of libxml2. Use "which xml2-config" to verify that the
program can be found where expected.
* Zlib Compression
The Zlib compression library is commonly installed on most systems
and is recommended for use with Swish-e. Zlib is used for
compressing text stored in the Swish-e index.
http://www.gzip.org/zlib/
Zlib must be installed before building Swish-e.
* Perl Modules
Although Swish-e is a compiled C program, many support features use
Perl. For example, both the web spiders and modules to help with
filtering documents are written in Perl.
The following Perl modules may be required. Check your current Perl
installation, as many may already be installed.
LWP
URI
HTML::Parser
HTML::Tagset
MIME::Types (optional)
Note that installing "Bundle::LWP" with the CPAN module
perl -MCPAN -e 'install Bundle::LWP'
will install many of the above modules.
If you wish to use "HTML-Template" with swish.cgi to generate
output, install:
HTML::Template
HTML::FillInForm
If you wish to use "Template-Toolkit" with "swish.cgi" to generate
output, install:
Template
Questions about installing these modules may be sent to the Swish-e
discussion list.
The "search.cgi" example script requires both "Template-Toolkit" and
"HTML::FillInForm".
* Indexing PDF Documents
Indexing PDF files requires the "xpdf" package. This is a common
package, available with most operating systems and often provided as
an add-on package.
http://www.foolabs.com/xpdf/
Xpdf may be added after Swish-e is installed.
* Indexing MS Word Documents
Indexing MS Word documents requires the Catdoc program.
http://www.wagner.pp.ru/~vitus/software/catdoc/
Catdoc may be added after Swish-e is installed.
* Indexing MP3 ID3 Tags
Indexing MP3 ID3 Tags requires the "MP3::Tag" Perl module. See
http://search.cpan.org. "MP3::Tag" may be installed after Swish-e is
installed.
* Indexing MS Excel Files
Indexing MS Excel files is supported by the following Perl modules,
also available at http://search.cpan.org.
Spreadsheet::ParseExcel
HTML::Entities
These Perl modules may be installed after Swish-e is installed.
INSTALLATION
Here are brief installation instructions that should work in most cases.
Following this section are more detailed instructions and examples.
Building Swish-e
Download Swish-e using your favorite web browser or a utility such as
"wget", "lynx", or "lwp-download". Unpack and build the distribution,
using the following steps:
Note: "swish-e-2.4.0" is used as an example. Download the most current
available version and adjust the commands below! Also, if you are
running Debian, see the notes below on building a ".deb" package from
the Swish-e source package.
Pay careful attention to the "prompt" character used on the following
command lines. A "$" prompt indicates steps run as an unprivileged user.
A "#" indicates steps run as the superuser (root).
$ wget http://swish-e.org/Download/swish-e-2.4.0.tar.gz
$ gzip -dc swihs-e-2.4.0.tar.gz | tar xof -
$ cd swish-e-2.4.0 (this directory will depend on the version of Swish-e)
$ ./configure
$ make
$ make check
...
==================
All 3 tests passed
==================
$ su root (or use sudo)
(enter password)
# make install
# exit
$ swish-e -V
SWISH-E 2.4.0
IMPORTANT: Once Swish-e is installed, do not run it as the superuser
(root) -- root is only required during the installation step, when
installing into system directories. Please do not break this rule.
NOTE: If you are upgrading from an older version of Swish-e, be sure and
review the CHANGES file. Old index files may not be compatible with
newer versions of Swish-e. After building Swish-e (but before running
"make install"), Swish-e can be run from the build directory:
$ src/swish-e -V
To minimize downtime, create new index files before running "make
install", by using Swish-e from the build directory. Then, copy the
index files to the live location and run "make install":
$ src/swish-e -c /path/to/config -f index.new
Keep in mind that the location you index from may affect the paths
stored in the index file.
Installing without root access
Here's another installation example. This might be used if you do not
have root access or you wish to install Swish-e someplace other than
"/usr/local".
This example also shows building Swish-e in a "build" directory that is
separate from where the source files are located. This is the
recommended way to build Swish-e, but it requires GNU Make. Without GNU
Make, you will likely need to build from within the source directory, as
shown in the previous example.
$ tar zxof swish-e-2.4.0.tar.gz (GNU tar with "z" option)
$ mkdir build
$ cd build
Note that the current directory is not where Swish-e was unpacked.
Swish-e uses a configure script. configure has many options, but it uses
reasonable and standard defaults. Running
$ ../swish-e-2.4.0/configure --help
will display the options.
Two options are of common interest: "--prefix" sets the top-level
installation directory; "--disable-shared" will link Swish-e statically,
which may be needed on some platforms (Solaris 2.6, perhaps).
Platforms may require varying link instructions when libraries are
installed in non-standard locations. Swish-e uses the GNU autoconf tools
for building the package. autoconf is good at building and testing, but
still requires you to provide information appropriate for your platform.
This may mean reading the manual page for your compiler and linker to
see how to specify non-standard file locations.
For most Unix-type platforms, you can use "LDFLAGS" and "CPPFLAGS"
environment variables to specify paths to "include" (header) files and
to libraries that are not in standard locations.
In this example, we do not have root access. We have installed libxml2
and libz in "$HOME/local". Swish-e will also be installed in
"$HOME/local" (by using the "--prefix" setting).
In this case, you would need to add "$HOME/local/bin" to the start of
your shell's $PATH setting. This is required because libxml2 installs a
program that is used when running the configure script. Before running
configure, type:
$ which xml2-config
It should list "$HOME/local/bin/xml2-config".
Now run configure (remember, we are in a separate "build" directory):
$ ../swish-e-2.4.0/configure \
--prefix=$HOME/local \
CPPFLAGS=-I$HOME/local/include \
LDFLAGS="-R$HOME/local/lib -L$HOME/local/lib"
$ make >/dev/null (redirect output to only see warnings and errors)
$ make check
...
==================
All 3 tests passed
==================
$ make install
$ $HOME/local/bin/swish-e -V
SWISH-E 2.4.0
Note the use of double quotes in the "LDFLAGS" line above. This allows
$HOME to be expanded within the text string.
Run-time paths
The "-R" option says to add a specified path (or paths) to those that
are used to find shared libraries at run time. These paths are stored in
the Swish-e binary. When Swish-e is run, it will look in these
directories for shared libraries.
Some platforms may not support the "-R" option. In this event, set the
"LD_RUN_PATH" environment variable before running make.
Some systems, such as Redhat, do not look in "/usr/local/lib" for
libraries. In these cases, you can either use "-R", as above, when
building Swish-e or add "/usr/local/lib" to "/etc/ld.so.conf" and run
ldconfig as root.
If all else fails, you may need to actually read the man pages for your
platform.
Building a Debian Package
The Swish-e distribution includes the files required to build a Debian
package.
$ tar zxof swish-e-2.4.0.tar.gz (GNU tar with "z" option)
$ cd swish-e-2.4.0
$ fakeroot debian/rules binary
[lots of output]
dpkg-deb: building package `swish-e' in `../swish-e_2.4.0-0_i386.deb'.
$ su
# dpkg -i ../swish-e_2.4.0-0_i386.deb
What's installed
Swish installs a number of files. By default, all files are installed
below "/usr/local", but this can be changed by setting "--prefix" when
running configure (as shown above). Individual paths may also be set.
Run "configure --help" for details.
$prefix/bin/swish-e The Swish-e binary program
$prefix/share/doc/swish-e/ Full documentation and examples
$prefix/lib/libswish-e The Swish-e C library
$prefix/include/swish-e.h The library header file
$prefix/man/man1/ Documentation as manual pages
$prefix/lib/swish-e/ Helper programs (spider.pl, swishspider, swish.cgi)
$prefix/lib/swish-e/perl/ Perl helper modules
Note that the Perl modules are *not* installed in the system Perl
library. Swish-e and the Perl scripts that require the modules know
where to find the modules, but the perldoc program (used for reading
documentation) does not. This can be corrected by adding
"$prefix/lib/swish-e" and "$prefix/lib/swish-e/perl" to the "PERL5LIB"
environment variable.
Documentation
Documentation can be found in the "$prefix/share/doc/swish-e" directory.
Documentation is in html format at "$prefix/share/doc/swish-e/html" and
can also be read on-line at the Swish-e web site:
http://swish-e.org/
The Swish-e documentation as man(1) pages
Running "make install" installs some of the Swish-e documentation as man
pages. The following man pages are installed:
SWISH-FAQ(1)
SWISH-CONFIG(1)
SWISH-RUN(1)
SWISH-LIBRARY(1)
The man pages are installed, by default, in the system man directory.
This directory is determined when configure is run; it can be set by
passing a directory name to configure.
For example,
./configure --mandir=/usr/local/doc/man
The man directory is specified relative to the "--prefix" setting. If
you use "--prefix", you do not normally need to also specify "--mandir".
Information on running configure can be found by typing:
./configure --help
Join the Swish-e discussion list
The final step, when installing Swish-e, is to join the Swish-e
discussion list.
The Swish-e discussion list is the place to ask questions about
installing and using Swish-e, see or post bug fixes or security
announcements, and offer help to others. Please do not contact the
developers directly.
The list is typically *very low traffic*, so it won't overload your
inbox. Please take the time to subscribe. See http://Swish-e.org.
If you are using Swish-e on a public site, please let the list know, so
that your URL can be added to the list of sites that use Swish-e!
Please review the next section before posting questions to the Swish-e
list.
QUESTIONS AND TROUBLESHOOTING
Support for installation, configuration, and usage is available via the
Swish-e discussion list. Visit http://swish-e.org for information. Do
not contact developers directly for help -- always post your question to
the list.
It's very important to provide the right information when asking for
help.
Please search the Swish-e list archive before posting a question. Also,
check the SWISH-FAQ to see if your question has already been asked and
answered.
Before posting, use the available tools to narrow down the problem.
Swish-e has several switches (e.g., "-T", "-v", and "-k") that may help
you resolve issues. These switches are described on the SWISH-RUN page.
For example, if you cannot find a document by a keyword that you believe
should be indexed, try indexing just that single file and use the "-T
INDEXED_WORDS" option to see if the word is actually being indexed.
First, try it without any changes to default settings:
swish-e -i testdoc.html -T indexed_words | less
if that works, add in your configuration file:
swish-e -i testdoc.html -c swish.conf -T indexed_words | less
If it still isn't working as you expect, try to reduce the test document
to a very small example. This will be very helpful to your readers, when
you are asking for help.
Another useful trick is to use "-H9" when searching, to display full
headers in search results. Look at the "Parsed Words" header to see what
words Swish-e is searching for.
When posting, please provide the following information:
Use these guidelines when asking for help. The most important tip is to
provide the least amount of information that can be used to reproduce
your problem. Do not paraphrase output -- copy-and-paste -- but trim
text that is not necessary.
* The exact version of Swish-e that you are using. Running Swish-e
with the "-V" switch will print the version number. Also, supply the
output from "uname -a" or similar command that identifies the
operating system you are running on. If you are running an old
version of swish, be prepared for a response of "upgrade" to your
question.
* A summary of the problem. This should include the commands issued
(e.g. for indexing or searching) and their output, along with an
explanation of why you don't think it's working correctly. Please
copy-and-paste the exact commands and their output, instead of
retyping, to avoid errors.
* Include a copy of the configuration file you are using, if any.
Swish-e has reasonable defaults, so in many cases you can run it
without using a configuration file. But, if you need to use a
configuration file, reduce it down to the absolute minimum number of
commands that is required to demonstrate your problem. Again,
copy-and-paste.
* A small copy of a source document that demonstrates the problem.
If you are having problems spidering a web server, use lwp-download
or wget to copy the file locally, then make sure you can index the
document using the file system method. This will help you determine
if the problem is with spidering or indexing.
If you expect help with spidering, don't post fake URLs, as it makes
it impossible to test. If you don't want to expose your web page to
the people on the Swish-e list, find some other site to test
spidering on. If that works, but you still cannot spider your own
site, you may need to request help from others. If so, you must post
your real URL or make a test document available via some other
source.
* If you are having trouble building Swish-e, please copy-and-paste
the output from make (or from "./configure", if that's where the
problem is).
The key is to provide enough information so that others may reproduce
the problem.
ADDITIONAL INSTALLATION OPTIONS
These steps are not required for normal use of Swish-e.
The SWISH::API Perl Module
The Swish-e distribution includes a module that provides a Perl
interface to the Swish-e C library. This module provides a way to search
a Swish-e index without running the Swish-e program. Searching an index
will be many times faster when running under a persistent environment
such as Apache/mod_perl with the "SWISH::API" module.
See the perl/README file for information on installing and using the
"SWISH::API" Perl module.
GENERAL CONFIGURATION AND USAGE
This section should give you a basic overview of indexing and searching
with Swish-e. Other examples can be found in the "conf" directory; these
will step you through a number of different configurations. Also, please
review the SWISH-FAQ.
Swish-e is a command-line program. The program is controlled by passing
switches on the command line. A configuration file may be used, but
often is not required. Swish-e does not include a graphical user
interface. Example CGI scripts are provided in the distribution, but
they require additional setup to use.
Introduction to Indexing and Searching
Swish-e can index files that are located on the local file system. For
example, running:
swish-e -i /var/www/htdocs
will index *all* files in the "/var/www/htdocs" directory. You may
specify one or more files or directories with the "-i" option. By
default, this will create an index called "index.swish-e" in the current
directory.
To search the resulting index for a given word, try:
swish-e -w apache
This will find the word "apache" in the body or title of the indexed
documents.
As mentioned above, Swish-e will index all files in a directory, unless
instructed otherwise. So, if "/var/www/htdocs" contains non-HTML files,
you will need a configuration file to limit the files that Swish-e
indexes. Create a file called "swish.conf":
# Example configuration file
# Tell Swish-e what to index (same as -i switch above)
IndexDir /var/www/htdocs
# Only index HTML and text files
IndexOnly .htm .html .txt
# Tell Swish-e that .txt files are to use the text parser.
IndexContents TXT* .txt
# Otherwise, use the HTML parser
DefaultContents HTML*
# Ask libxml2 to report any parsing errors and warnings or
# any UTF-8 to 8859-1 conversion errors
ParserWarnLevel 9
After saving the configuration file, reindex:
swish-e -c swish.conf
The Swish-e configuration settings are described in the SWISH-CONFIG
manual page. The order of statements in the configuration file is
typically not important, although some statements depend on previously
set statements. There are many possible settings. Good advice is to use
as few settings as possible when first starting out with Swish-e.
The runtime options (switches) are described in the SWISH-RUN manual
page. You may also see a summary of options by running:
swish-e -h
Swish-e has two other methods for reading input files. One method uses a
Perl helper script and the LWP Perl library to spider remote web sites:
swish-e -S http -i http://localhost/index.html -v2
This will spider the web server running on the local host. The "-S"
option defines the input source method to be "http", "-i" specifies the
URL to spider, and "-v" sets the verbose level to two. There are a
number of configuration options that are specific to the "-S" http input
source. See SWISH-CONFIG. Note that only files of "Content-Type text/*"
will be indexed.
The "-S http" method is deprecated, however, in favor of a variation on
the following input method.
There is a general-purpose input method wherein Swish-e reads input from
a program that produces documents in a special format. The program might
read and format data stored in a database, or parse and format messages
in a mailing list archive, or run a program that spiders web sites (like
the previous method).
The Swish-e distribution includes a spider program that uses this method
of input. This spider program is much more configurable and feature-rich
than the previous ("-S http") method.
To duplicate the previous example, create a configuration file called
"swish2.conf":
# Example for spidering
# Use the "spider.pl" program included with Swish-e
IndexDir spider.pl
# Define what site to index
SwishProgParameters default http://localhost/index.html
Then, create the index using the command:
swish-e -S prog -c swish2.conf
This says to use the "-S prog" input source method. Note that, in this
case, the "IndexDir" setting does not specify a file or directory to
index, but a program name to be run. This program, "spider.pl", does the
work of fetching the documents from the web server and passing them to
Swish-e for indexing.
The "SwishProgParameters" option is a special feature that allows
passing command-line parameters to the program specified with
"IndexDir". In this case, we are passing the word "default" (which tells
"spider.pl" to use default settings) and the URL to spider.
Running a script under Windows requires specifying the interpreter
(e.g., "perl.exe") and then using "SwishPropParameters" to specify the
script and the script's parameters. See *Notes when using "-S prog" on
MS Windows* on the SWISH-RUN page.
The advantage of the "-S prog" method of spidering (over the previous
"-S http" method) is that the Perl code is only compiled once instead of
once for every document fetched from the web server. In addition, it is
a much more advanced spider with many, many features. Still, as used
here, "spider.pl" will automatically index PDF or MS Word documents if
(when) Xpdf and Catdoc are installed.
A special form of the "-S prog" input source method is:
./myprog --option | swish-e -S prog -i stdin -c config
This allows running Swish-e from a program (instead of running the
external program from Swish-e). So, this also can be done as:
./myprog --option > outfile
swish-e -S prog -i stdin -c config < outfile
or
./myprog --option > outfile
cat outfile | swish-e -S prog -i stdin -c config
One final note about the "-S prog" input source method. The program
specified with "-i" or "IndexDir" needs to be an absolute path. The
exception is when the program is installed in the "libexecdir"
directory. Then, a plain program name may be specified (as in the
example showing "spider.pl", above).
All three input source methods are described in more detail on the
SWISH-RUN page.
Metanames and Properties
There are two key Swish-e concepts that you need to be familiar with:
Metanames and Properties.
* Metanames
Swish-e creates a reverse (i.e., inverted) index. Just like an index
in a book, you look up a word and it lists the pages (or documents)
where that word can be found.
Swish-e can create multiple index tables within the same index file.
For example, you might want to create an index that only contains
words in HTML titles, so that searches can be limited to title text.
Or, you might have descriptive words that you would like to search,
stored in a meta tag called "keywords".
Some database systems might call these different "fields" or
"columns", but Swish-e calls them *MetaNames* (as a result of its
first indexing HTML "meta" tags).
To find documents containing "foo" in their titles, you might run:
swish-e -w swishtitle=foo
or, a more advanced example:
swish-e -w swishtitle=(foo or bar) or swishdefault=(baz)
The Metaname "swishdefault" is the name that is used by Swish-e if
no other name is specified. The following two searches are thus
equivalent:
swish-e -w foo
swish-e -w swishdefault=foo
When indexing HTML documents, Swish-e indexes words in the body and
title under the Metaname "swishdefault".
* Properties
Swish-e's search result is a list of files -- actually, Swish-e uses
file numbers internally. Data can be associated with each file
number when indexing. For example, by default Swish-e associates the
file's name, title, last modified date, and size with the file
number. These items can be printed in search results.
In Swish-e, this associated data is called a file's *Properties*.
Properties can be any data you wish to associated with a document --
in fact, the entire text of the document can be stored in the index.
What data is stored as a Property is controlled by the
*PropertyNames* (and other) configuration directives.
What properties are printed with search results depends on the "-x"
or "-p" switches. By default, Swish-e returns the rank, path/URL,
title, and file size in bytes for each result.
Getting Started With Swish-e
Swish-e reads a configuration file (see SWISH-CONFIG) for directives
that control whether and how Swish-e indexes files. Swish-e is also
controlled by command-line arguments (see SWISH-RUN). Many of the
command-line arguments have equivalent configuration directives (e.g.,
"-i" and "IndexDir").
Swish-e does not require a configuration file, but most people change
its default behavior by placing settings in a configuration file.
To try the examples below, go to the "tests" subdirectory of the
distribution. The tests will use the "*.html" files in this directory
when creating the test index. You may wish to review these "*.html"
files to get an idea of the various native file formats that Swish-e
supports.
You may also use your own test documents. It's recommended to use small
test documents when first using Swish-e.
Step 1: Create a Configuration File
The configuration file controls what and how Swish-e indexes. The
configuration file consists of directives, comments, and blank lines.
The configuration file can be any name you like.
This example will work with the documents in the tests directory. You
may wish to review the tests/test.config configuration file used for the
"make test" tests.
For example, a simple configuration file (swish-e.conf):
# Example Swish-e Configuration file
# Define *what* to index
# IndexDir can point to a directories and/or a files
# Here it's pointing to the current directory
# Swish-e will also recurse into sub-directories.
IndexDir .
# But only index the .html files
IndexOnly .html
# Show basic info while indexing
IndexReport 1
And that's a simple configuration file. It says to index all the ".html"
files in the current directory and sub-directories, if any, and provide
some basic output while indexing.
As mentioned above, the complete list of all configuration file
directives is detailed in SWISH-CONFIG.
Step 2: Index your Files
Run Swish-e, using the "-c" switch to specify the name of the
configuration file.
swish-e -c swish-e.conf
Indexing Data Source: "File-System"
Indexing "."
Removing very common words...
no words removed.
Writing main index...
Sorting words ...
Sorting 55 words alphabetically
Writing header ...
Writing index entries ...
Writing word text: Complete
Writing word hash: Complete
Writing word data: Complete
55 unique words indexed.
4 properties sorted.
5 files indexed. 1252 total bytes. 140 total words.
Elapsed time: 00:00:00 CPU time: 00:00:00
Indexing done!
This created the index file "index.swish-e". This is the default index
file name, unless the IndexFile directive is specified in the
configuration file:
IndexFile ./website.index
You may use the "-f" switch to specify a index file at indexing time.
The "-f" option overrides any "IndexFile" setting that may be in the
configuration file.
Step 3: Search
You specify your search terms with the "-w" switch. For example, to find
the files that contain the word "sample", you would issue the command:
swish-e -w sample
This example assumes that you are in the "tests" directory. Swish-e
returns the following, in response to this command:
swish-e -w sample
# SWISH format: 2.4.0
# Search words: sample
# Number of hits: 2
# Search time: 0.000 seconds
# Run time: 0.005 seconds
1000 ./test_xml.html "If you are seeing this, the METATAG XML search was successful!" 159
1000 ./test.html "If you are seeing this, the test was successful!" 437
.
So, the word "sample" was found in two documents. The first number shown
is the relevance (or rank) of the search term, followed by the file
containing the search term, the title of the document, and finally, the
length of the document (in bytes).
The period ("."), sitting alone at the end, marks the end of the search
results.
Much more information may be retrieved while searching, by using the
"-x" and "-H" switches (see SWISH-RUN) and by using Document Properties
(see SWISH-CONFIG).
Phrase Searching
To search for a phrase in a document, use double-quotes to delimit your
search terms. (The default phrase delimiter is set in "src/swish.h".)
You must protect the quotes from the shell.
For example, under Unix:
swish-e -w '"this is a phrase" or (this and that)'
swish-e -w 'meta1=("this is a phrase") or (this and that)'
Or under the Windows "command.com" shell.
swish-e -w \"this is a phrase\" or (this and that)
The phrase delimiter can be set with the "-P" switch.
Boolean Searching
You can use the Boolean operators and, or, or not in searching. Without
these Boolean operatots, Swish-e will assume you're anding the words
together.
Here are some examples:
swish-e -w 'apples oranges'
swish-e -w 'apples and oranges' ( Same thing )
swish-e -w 'apples or oranges'
swish-e -w 'apples or oranges not juice' -f myIndex
retrieves first the files that contain both the words "apples" and
"oranges"; then among those, selects the ones that do not contain the
word "juice".
A few other examples to ponder:
swish-e -w 'apples and oranges or pears'
swish-e -w '(apples and oranges) or pears' ( Same thing )
swish-e -w 'apples and (oranges or pears)' ( Not the same thing )
Swish processes the query left to right.
See SWISH-SEARCH for more information.
Context Searching
The "-t" option in the search command line allows you to search for
words that exist only in specific HTML tags. This option takes a string
of characters as its argument. Each character represents a different tag
in which the word is searched; that is, you can use any combinations of
the following characters:
H search in all tags
B search in the tags
t search in tags
h is <H1> to <H6> (header) tags
e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>)
c is HTML comment tags (<!-- ... -->)
For example:
# Find only documents with the word "linux" in the <TITLE> tags.
swish-e -w linux -t t
# Find the word "apple" in titles or comments
swish-e -w apple -t tc
META Tags
As mentioned above, Metanames are a way to define "fields" in your
documents. You can use the Metanames in your queries to limit the search
to just the words contained in that META name of your document. For
example, you might have a META-tagged field called "subjects" in your
documents. This would let you search your documents for the word "foo",
but only return documents where "foo" is within the "subjects" META tag.
Document *Properties* are somewhat related: Properties allow the content
of a META tag in a source document to be stored within the index, and
that text to be returned along with search results.
META tags can have two formats in your documents.
<META NAME="keyName" CONTENT="some Content">
And in XML format
<keyName>
Some Content
</keyName>
If using libxml, you can optionally use a non-HTML tag as a metaname:
<html>
<body>
Hello swish users!
<keyName>
this is meta data
</keyName>.
</body>
This, of course, is invalid HTML.
To continue with our sample "Swish-e.conf" file, add the following
lines:
# Define META tags
MetaNames meta1 meta2 meta3
Reindex to include the changes:
swish-e -c swish-e.conf
Now search, but this time limit your search to META tag "meta1":
swish-e -w 'meta1=metatest1'
Again, please see SWISH-RUN and SWISH-CONFIG for complete documentation
of the various indexing and searching options.
Spidering and Searching with a Web form.
This example demonstrates how to spider a web site and set up the
included CGI script to provide a web-based search page. This example
uses Perl programs that are included in the Swish-e distribution:
spider.pl will be used for reading files from the web server; swish.cgi
will provide the web search form and display results.
As an example, we will index the Apache Web Server documentation,
installed on the local computer at
http://localhost/apache_docs/index.html.
1 Make a Working Directory
Create a directory to store the Swish-e configuration and the
Swish-e index.
~$ mkdir web_index
~$ cd web_index/
~/web_index$
2 Create a Swish-e Configuration file
~/web_index$ cat swish.conf
# Swish-e config to index the Apache documentation
#
# Use spider.pl for indexing (location of spider.pl set at installation time)
IndexDir spider.pl
# Use spider.pl's default configuration and specify the URL to spider
SwishProgParameters default http://localhost/apache_docs/index.html
# Allow extra searching by title, path
Metanames swishtitle swishdocpath
# Set StoreDescription for each parser
# to display context with search results
StoreDescription TXT* 10000
StoreDescription HTML* <body> 10000
3 Generate the Index
Now, run Swish-e to create the index:
~/web_index$ swish-e -S prog -c swish.conf
Indexing Data Source: "External-Program"
Indexing "spider.pl"
/usr/local/lib/swish-e/spider.pl: Reading parameters from 'default'
Summary for: http://localhost/apache_docs/index.html
Duplicates: 4,188 (349.0/sec)
Off-site links: 276 (23.0/sec)
Skipped: 1 (0.1/sec)
Total Bytes: 2,090,125 (174177.1/sec)
Total Docs: 147 (12.2/sec)
Unique URLs: 149 (12.4/sec)
Removing very common words...
no words removed.
Writing main index...
Sorting words ...
Sorting 7736 words alphabetically
Writing header ...
Writing index entries ...
Writing word text: Complete
Writing word hash: Complete
Writing word data: Complete
7736 unique words indexed.
5 properties sorted.
147 files indexed. 2090125 total bytes. 200783 total words.
Elapsed time: 00:00:13 CPU time: 00:00:02
Indexing done!
The above output is actually a mix of output from both Swish-e and
"spider.pl". "spider.pl" reports the "Summary for:
http://localhost/apache_docs/index.html".
Also note that Swish-e knows to find "spider.pl" at
"/usr/local/lib/swish-e/spider.pl". The script installation
directory (called "libexecdir") is set at configure time. You can
see your setting by running "swish-e -h":
~/web_index$ swish-e -h | grep libexecdir
Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e
This directory will be needed in the next step, when setting up the
CGI script.
Finally, verify that the index can be searched from the command
line:
~/web_index$ swish-e -w installing -m3
# SWISH format: 2.4.0
# Search words: installing
# Removed stopwords:
# Number of hits: 17
# Search time: 0.018 seconds
# Run time: 0.050 seconds
1000 http://localhost/apache_docs/install.html "Compiling and Installing Apache" 17960
718 http://localhost/apache_docs/install-tpf.html "Installing Apache on TPF" 25734
680 http://localhost/apache_docs/windows.html "Using Apache with Microsoft Windows" 27165
.
Now, try limiting the search to the title:
~/web_index$ swish-e -w swishtitle=installing -m3
# SWISH format: 2.3.5
# Search words: swishtitle=installing
# Removed stopwords:
# Number of hits: 2
# Search time: 0.018 seconds
# Run time: 0.048 seconds
1000 http://localhost/apache_docs/install-tpf.html "Installing Apache on TPF" 25734
1000 http://localhost/apache_docs/install.html "Compiling and Installing Apache" 17960
.
Note that the above can also be done using the "-t" option:
~/web_index$ swish-e -w installing -m3 -tH
4 Set up the CGI script
Swish-e does not include a web server. So, you must use your locally
installed web server. Apache is highly recommended, of course.
Locate your web server's CGI directory. This may be a "cgi-bin"
directory in your home directory or a central "cgi-bin" directory
set up by the web server administrator. Once this is located, copy
the "swish.cgi" script into the "cgi-bin" directory.
Where CGI scripts can be located depends completely on the web
server that is being used and how it has been configured. See your
web server's documentation or your site's administrator for
additional information.
This example will use a site "cgi-bin" directory, located at
"/usr/lib/cgi-bin". Copy the "swish.cgi" script into the "cgi-bin"
directory. Again, we will need the location of the "libexecdir"
directory:
~/web_index$ swish-e -h | grep libexecdir
Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e
~/web_index$ cd /usr/lib/cgi-bin
/usr/lib/cgi-bin$ su
Password:
/usr/lib/cgi-bin# cp /usr/local/lib/swish-e/swish.cgi.
If your operating system supports symbolic links and your web server
allows programs to be symbolic links, then you may wish to create a
link to the "swish.cgi" program, instead.
/usr/lib/cgi-bin# ln -s /usr/local/lib/swish-e/swish.cgi
We need to tell the "swish.cgi" script where to look for the index
created in the previous step. It's also recommended to enter the
path to the swish-e binary. Otherwise, the "swish.cgi" script will
look for the binary in the "PATH", and that may change when running
under the CGI environment.
Here's the configuration file:
/usr/lib/cgi-bin# cat .swishcgi.conf
return {
title => 'Search Apache Documentation',
swish_binary => '/usr/local/bin/swish-e',
swish_index => '/home/moseley/web_index/index.swish-e',
}
Now, test the script from the command line (as a normal user!):
/usr/lib/cgi-bin# exit
exit
/usr/lib/cgi-bin$ ./swish.cgi | head
Content-Type: text/html; charset=ISO-8859-1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>
Search Apache Documentation
Notice that the CGI script returns the HTTP header (Content-Type)
and the body of the web page, just like a well behaved CGI scrip
should do.
Now, test using the web server (this step depends on the location of
your "cgi-bin" directory). This example uses the "GET" command that
is part of the LWP Perl library, but any web browser can run this
test.
/usr/lib/cgi-bin$ GET http://localhost/cgi-bin/swish.cgi | head
Search Apache Documentation
The script reports errors to stderr, so consult the web server's
error log if problems occur. The message "Service currently
unavailable", reported by running "swish.cgi", typically indicates a
configuration error; the exact problem will be listed in the web
server's error log.
Detailed instructions on using the "swish.cgi" script and debugging
tips can be found by running:
$ perldoc swish.cgi
while in the "cgi-bin" directory where "swish.cgi" was copied.
The spider program "spider.pl" also has a large number of
configuration options.
Documentation is also available in the directory
"$prefix/share/doc/swish-e" or at http://swish-e.org.
Note: Also check out the "search.cgi" script, found at the same
location as the "swish.cgi" script. This is more of a skeleton
script, for those that want to create a custom search script.
Now you are ready to search.
Indexing Other Types of Documents - Filtering
Swish-e can only index HTML, XML, and text documents. In order to index
other documents, such as PDF or MS Word documents, you must use a
utility to convert or "filter" those documents.
How documents are filtered with Swish-e has changed over time. This has
resulting in a bit of confusion. It's also a somewhat complex process,
as different programs need to communicate with each other.
You may wish to read the Swish-e FAQ question on filtering, before
continuing here. How do I filter documents?
Filtering Overview
There are two ways to filter documents with Swish-e. Both are described
in the SWISH-CONFIG man page. They use the "FileFilter" directive and
the "SWISH::Filter" Perl module.
The "FileFilter" directive is a general-purpose method of filtering. It
allows running of an external program for each document processed (based
on file extension), and requires one or more external programs. These
programs open an input file, convert as needed, and write their output
to standard output.
Previous versions of Swish-e (before 2.4.0) used a collection of filter
programs for converting files such as PDF or MS Word documents. The
external programs call other program to do the work of filtering (e.g.
pdftotext to extract the contents from PDF files). Although these filter
programs are still included with the Swish-e distribution as examples,
it is recommended to use the "SWISH::Filter" method, instead.
One disadvantage of using "FileFilter" is that the filter program is run
once for every document that needs to be filtered. This can slow down
the indexing process substantially.
The "SWISH::Filter" Perl module works very much like the old system and
uses the same helper programs. Convieniently, however, it provides a
single interface for filtering all types of documents. The primary
advantage of "SWISH::Filter" is that it is built into the program used
for spidering web sites (spider.pl), so all that's required is
installing the filter programs that do the actual work of filtering
(e.g. catdoc, xpdf). (The Windows binary includes some of the filter
programs.)
But, Swish-e will not use "SWISH::Filter" by default when using the file
system method of indexing. To use "SWISH::Filter" when indexing by file
system method (-S fs), you can use a "FileFilter" directive with the
"swish_filter.pl" filter (which is just a program that uses
"SWISH::Filter") or use the "-S prog" method of indexing and use the
"DirTree.pl" program for fetching documents.
"DirTree.pl" is included with the Swish-e distribution and is designed
to work with "SWISH::Filter". Using DirTree.pl will likely be a faster
way to index, since the "SWISH::Filter" set of modules does not need to
be compiled for every document that needs to be filtered.
See the contents of "swish_filter.pl" and "DirTree.pl" for specifics on
their use.
Filtering Examples
The "FileFilter" directive can be used in your config file to convert
documents, based on their extensions. This is the old way of filtering,
but provides an easy way to add filters to Swish-e.
For example:
FileFilter .pdf pdftotext "'%p' -"
IndexContents TXT* .pdf
will cause all ".pdf" files to be filtered through the pdftotext program
(part of the Xpdf package) and to parse the resulting output (from
pdftotext) with the text ("TXT") parser.
The other way to filter documents is to use a "-S prog" prograam and
convert the documents before passing them onto Swish-e.
For example, "spider.pl" makes use of the "SWISH::Filter"" Perl module,
included with the Swish-e distribution. "SWISH::Filter" is passed a
document and the document's content type; it looks for modules and
utilities to convert the document into one of the types that Swish-e can
index.
Swish-e comes ready to index PDF, MS Word, MP3 ID3 tags, and MS Excel
file types. But these filters need extra modules or tools to do the
actual conversion.
For example, the Swish-e distribution includes a module called
"SWISH::Filter::Pdf2HTML" that uses the pdftotext and pdfinfo utilities
provided by the Xpdf package.
This means that if you are using "spider.pl" to spider your web site and
you wish to index PDF documents, all that is needed is to install the
Xpdf package and Swish-e (with the help of spider.pl) will begin
indexing your PDF files.
Ok, so what does all that mean? For a very simple site, you should be
able to run this:
$ /usr/local/lib/swish-e/spider.pl default http://localhost/ | swish-e -S prog -i stdin
which is running the spider with default spider settings, indexing the
Web server on localhost, and piping its output into Swish-e (using the
default indexing settings). Documents will be filtered automatically, if
you have the required helper applications installed.
Most people will not want to just use the default settings (for one
thing, the spider will take a while because its default is to delay a
few seconds between every request). So, read the documentation for
"spider.pl", to learn how to use a spider config file. Also read
SWISH-CONFIG to learn about what configuration options can be used with
Swish-e.
The "SWISH::Filter" documentation provides more details on filtering and
hints for debugging problems when filtering.
Document Info
$Id: INSTALL.pod 1978 2007-12-08 01:59:17Z karpet $
.