Module Ufind

For enforcing a case-sensitive search, just use CF_NONE. The casefolding function can also be used to normalize the utf8 string; this is done by CF_D145 and CF_D147. However, recall that normalizing is a slow operation, and it should rather be done directly when storing items in your database. The expected properties of the casefolding function are:

• Removing accents (with utf8_to_ascii) followed by casefolding should give the same result as casefolding followed be removing accents. The resulting string is called the base string. It contains only ASCII characters.

• Applying casefolding on a base string should return the unmodified string.

• If two items have different base strings, they will never match in any search.

• If two items have the same base string, they should match in any search, but maybe we a low ranking (low matching quality).

Expected property for case-insensitive search:

• Equality of casefolded ASCII strings should be equivalent to equality of the strings obtained by applying String.lowercase_ascii.

CUSTOM casefolding functions can be used for specific cases. For instance, one can use capitalize_casefold, which combines a usual lower-case function with capitalizing the first letter. In this way we force the match to happen at the start of the string: "Ver" (or "ver") will match with "Véronique" and "VÉRONIQUE" but not with "Prévert" or "PRÉVERT". Of course, the same result can be obtained by a simple matching_defect function, see below.

Return the ASCII version of the item.

Return the data associated with the item.

preprocess ~get_name ~get_data seq returns a sequence of search_items from the source sequence seq.

This function is a general way of obtaining a sequence of search items from any kind of data source, as long as this source can be accessed by a sequence (ocaml type Seq.t). The function get_name takes an element of the sequence and should return the name field that we are searching. The function get_data on an element of the sequence can return any type of data that we want to associate with the result of the search. It can be the same as get_name. But, the data can also be the id of the correspondng record, so that from the result of the search on the name field we can recover the other fields of the matching records.

The preprocess function is not lazy; as a consequence, the resulting sequence is very fast to search. If limit=(offset,count), preprocess will force evaluation of the first offset+count elements of the sequence seq, and return a sequence of at most count effectively computed search_items starting at item #offset (inclusive). Warning, if no limit is given, the whole seq is processed; hence if seq is infinite, it will never terminate until memory overflows.

In all preprocess* functions, the returned sequence is not mutable, has no side-effect, and will always point to the start of the source sequence. So there is no need to reset it for each new search.

Same as preprocess but the database is a list instead of a sequence.

Use this to search in a file, where each item must be encoded in a single line.

The name field should be obtained by get_name line, and the data field by get_data line. Returns the sequence of pre-processed search items.

Similar to preprocess except that there is no preprocessing: this immediately returns a lazy sequence of items, suitable for searching.

If the source sequence is mutable (most sequences are), then for each new search, it has to be reset to its origin position, and a new call to items_from_seq is required.

If memory allows and if you intend to perform several searches, use preprocess for faster searching.

items_from_names list immediately returns a lazy sequence of search items from a list of strings.

The returned sequence is not mutable, and will always point to the start of the list. So there is no need to reset it for each new search.

For faster searching, rather use preprocess_list ~get_name:id ~get_data:id list, where id x = x. The only interest of items_from_names is when the list is really long and we don't want to duplicate it in memory.

items_from_text text immediately constructs a lazy list of search_item corresponding to each word of the string text, where word delimiters are [ \t\n()]. Usual punctuation signs are removed from the end of words.

The returned sequence is not mutable, and will always point to the start of the text. So there is no need to reset it for each new search.

After searching with select_data, the resulting data is a list of pairs (pos, word) where pos is the position of the word in the original string.

items_from_channel channel immediately constructs a lazy list of search_item corresponding to each word read from the channel, where word delimiters are [ \t\n()]. Usual punctuation signs are removed from the end of words.

The resulting sequence is mutable, and will point to the current search position in the channel, which is not closed by this function.

After searching with select_data, the resulting data is a list of pairs (pos, word) where pos is the byte position of the word in channel, starting from the initial state of the channel.

select_data items name searches for the string name within the sequence of search items items and returns the sorted list of data corresponding to matching items.

If stop is not provided, the search will explore the whole sequence. Otherwise, the search will stop after processing the first item in items for which stop item = true. The argument matching_stop operates in a similar way, but is executed only on matching items, and its argument is the couple (distance, item).

The folding parameter must be the same as the one used to create the items sequence.

make_stop ~count ~timeout () returns the pair (stop, flag), where stop is a 'stop' function suitable for use in select_data; and flag is a function that returns true when the stop test is effectively triggered.

When used with select_data, the search will stop after processing count elements, or when the timeout (in seconds) is elapsed. Note that the timer starts as soon at the unit argument () is provided.

The stop function has to be created again after each use.

In conjunction with seq_split, the flag can be used to resume a previously stopped search, as follows.

let seq1, seq2 = seq_split items in
  let stop, flag = make_stop ~count:10 () in
  let result = select_data ~stop seq1 name in
if flag () then begin
  print_endline "The search was interrupted. We resume it.";
  let result2 = select_data seq2 name in
  (* ...now the complete result is the union {result,result2}, 
        but the global ranking is lost... *)
end else print_endline "The search was complete."

This is the simplest search function; it doesn't require any preprocessing. filter_list ~get_name name list will filter (and sort by relevance) the given list by returning only those elements whose name field (extracted by get_name) matches name.

Example:

# let list = [1,"Arthur"; 2,"Benoît"; 3,"Camille"; 4,"Damián"] in
  filter_list ~get_name:snd "á" list;;

- : (int * string) list = [(4, "Dami\195\161n"); (1, "Arthur"); (3, "Camille")]

Because the returned list is just a subset of the initial list, one can easily refine a search by chaining several calls to filter_list.

Obtaining detailed matching results

Evaluate the whole sequence and convert it to a list, in reverse order.

Immediately return a lazy sequence from the given list.

Evaluate the whole sequence and return a new sequence with the evaluated values.

(Half)-immediate truncation of a sequence. seq_truncate offset count seq returns a sequence of length count (or less in case the initial sequence is too short) containing the elements of the initial seq starting at the offset-eth element.

This operation is not entirely lazy: elements before #offset will be evaluated. But no other element.

seq_stop stop seq returns a lazy truncation of the sequence seq. When evaluated, the sequence will stop just after the function stop evaluated on an element returns true. For instance stop can be a timer, see make_stop.

Dynamic splittig of a sequence. If s1,s2 = seq_split seq then s2 will always start after the last evaluated element of s1.

For instance if we let st = seq_truncate 0 10 s1 and iterate on st, then the iteration of s2 will start at the 11th element of seq.

Convert ISO_8859_1 to UTF8

Convert to ASCII by removing all accents on Latin letters, and removing all other non-ascii chars or non-Latin letters.

Example:

# utf8_to_ascii "¡Déjà vu!";;
- : string = "Deja vu!"