ascii text about fido HOT 5 CLOSED

Because file (and Tika) have additional heuristic format detection modules specifically for this case. IIRC, the file one works in the same way as the Tika one, and is based on the absence bytes other than those consistent with ASCII/UTF-8 in the first X bytes of the item.

DROID only has binary signature and container signature modules, and therefore so does Fido. It might be possible to add such a module to Fido as a fallback if binary and container signatures have no match. Of course, this would make the results inconsistent with those from DROID, so it would presumably have to be an opt-in option.

Also, I've noticed that Tika has deliberately decided to make the 'text/plain' detection take precedence over file-extension based identification unless the identified format is known to be a sub-set of 'text/plain'. I'm not sure what the rationale behind this is, but it might be worth finding out in case we've missed something. I suspect it's just because they are interested in making sure the text is extracted reliably, in which case we need not worry as that's not what DROID/Fido is for.

from fido.

Thanks @anjackson I guess the same question could be posed to DROID. It seems kind of wrong that a simple ASCII text file can't be identified. I was running fido on a set of born digital content, and the results looked like:

??? 6971
WordPerfect for MS-DOS Document 4379
WordPerfect for MS-DOS/Windows Document 1896
Microsoft Word for Windows Macro    76
OLE2 Compound Document Format   26
Plain Text File 25
PrimeOCR    25
Generic Library File    19
Comma Separated Values  17
Adobe Font List 11
MacPaint Image  8
Microsoft Office Owner File 6
Hypertext Markup Language   6
Lotus Notes File    3
Microsoft Word for Macintosh Document   3
PostScript  3
MS-DOS Executable   3
Microsoft Excel Backup  2
Encapsulated PostScript File Format 2
Microsoft Windows Cursor    2
Lotus 1-2-3 Spreadsheet Formatting File 2
Macromedia Director 2
AutoCAD Script  2
dBASE Database  2
Applixware Spreadsheet  2
Freelance File  2
LaTeX (Subdocument) 1
Microsoft Animated Cursor Format    1
Digital Terrain Elevation Data  1
DesignCAD Drawing   1
Windows New Executable  1
Batch file (executable) 1
Statistica Report File  1
Internet Message Format 1
Pagemaker TableEditor Graphics  1
Enigma Binary File (Finale) 1
Rocket Book eBook format    1
Microsoft Word for Windows Document 1
Cascading Style Sheet   1
Vista Pro Graphics  1
Macintosh PICT Image    1

Where a large number of ??? were ASCII text files which (in theory) should've been lumped in with Plain Text Files.

from fido.

I’m not sure of the current code, but previous versions of fido had an extension mechanism to cover this sort of thing – as well as various issues with pronom signatures at the time. Droid-compatibility is useful as an option, but doesn’t need to be either the default or only behaviour. Again, I’m not sure of the current code, but there has always a need to prioritise matches (e.g., we might prefer xml or python as types over generic utf-8), so simply adding an extension matching ascii and another to match utf-8 should be pretty easy.

from fido.

DROID does use extension-based matching, but I'm not sure how good the coverage is (there are an awful lot of text/plain types after all). This (along with the fact that DROID is a bit too strict with messy formats like HTML) is why I tend to use Tika and DROID in combination.

Which reminds me that none of these tools are very good at plain text formats unless the extensions is correct. Reliably detecting CSS, JS, CSV, etc. is still painful.

from fido.

Closing as seems to be a new feature release for text matching that's unlikely to be developed.

from fido.