Function: csv--separator-score

csv--separator-score is a natively compiled function defined in csv-mode.el.

Signature

(csv--separator-score SEPARATOR TEXT &optional CUTOFF)

Documentation

Return a score on how likely SEPARATOR is a separator in TEXT.

When CUTOFF is passed, stop the calculation at the next whole line after having read CUTOFF number of characters.

The scoring is based on the idea that most CSV data is tabular, i.e. separators should appear equally often on each line. Furthermore, more commonly appearing characters are scored higher than those who appear less often.

Adapted from the paper "Wrangling Messy CSV Files by Detecting Row and Type Patterns" by Gerrit J.J. van den Burg , Alfredo Nazábal, and Charles Sutton: https://arxiv.org/abs/1811.11242.

Source Code

;; Defined in /nix/store/3gmjnyw9n226b7k7jamikx980vf8267p-emacs-packages-deps/share/emacs/site-lisp/elpa/csv-mode-1.27/csv-mode.el
(defun csv--separator-score (separator text &optional cutoff)
  "Return a score on how likely SEPARATOR is a separator in TEXT.

When CUTOFF is passed, stop the calculation at the next whole
line after having read CUTOFF number of characters.

The scoring is based on the idea that most CSV data is tabular,
i.e. separators should appear equally often on each line.
Furthermore, more commonly appearing characters are scored higher
than those who appear less often.

Adapted from the paper \"Wrangling Messy CSV Files by Detecting
Row and Type Patterns\" by Gerrit J.J. van den Burg , Alfredo
Nazábal, and Charles Sutton: https://arxiv.org/abs/1811.11242."
  (let ((groups
         (with-temp-buffer
           (csv-set-separator separator)
           (save-excursion
             (insert text))
           (let ((groups (make-hash-table))
                 (chars-read 0))
             (while (and (/= (point) (point-max))
                         (or (not cutoff)
                             (< chars-read cutoff)))
               (let* ((lep (line-end-position))
                      (nfields (length (csv--collect-fields lep))))
                 (cl-incf (gethash nfields groups 0))
                 (cl-incf chars-read (- lep (point)))
                 (goto-char (+ lep 1))))
             groups)))
        (sum 0))
    (maphash
     (lambda (length num)
       (cl-incf sum (* num (/ (- length 1) (float length)))))
     groups)
    (let ((unique-groups (hash-table-count groups)))
      (if (= 0 unique-groups)
          0
        (/ sum unique-groups)))))