Skip to content

File types, kinds & parser limits

Every file DioscHub stores is given a kind when it arrives. The kind decides what happens next: whether the file is read and indexed, which parser reads it, whether it can reach the model as an image, and how it is served back. One built-in table drives all of it, and administrators can extend it with file type rules on System → File types.

This page covers the kinds and formats, how classification works, the rules, the limits that bound every parser, and how to re-classify files that are already stored. The upload size limit, storage quota and retention are on File storage limits, quota & retention.

KindWhat happens to the file
documentRead by its parser and indexed for search.
tableRead and indexed. XLSX, CSV and TSV files in a user’s library also get a workbook outline and can be queried with the table tools.
textRead as text and indexed, each chunk located by line numbers. Code, markup and data files are text.
imageStored and marked ready; no text is extracted. PNG, JPEG, WebP and GIF images can be sent to a vision model.
mediaAudio and video. Stored only, not read.
archiveZip, gzip, tar and the like. Stored only, not opened.
otherAnything DioscHub does not recognise, or whose bytes don’t match their claimed type. Stored only, and always served as a download.

A file that is stored only has the status stored_only with one of two reasons:

  • stored_only_kind — “This kind of file is stored but not read.” (media, archives and unknown files);
  • type_mismatch — “The file’s content doesn’t match its type.”

Files DioscHub derives for its own use have a fixed kind: derived for a cached table, and image for a picture extracted from a workbook. Neither is ever listed as one of the user’s files. Files a tool generates (generate_file, bundle_files) get the kind of the format the generator produced; rules are not applied to them.

The built-in table. System → File types shows the same list under Built-in types.

ExtensionParserWhat is indexed
.pdfpdfThe text layer, by page.
.docxdocxBody text, then headers, footers, footnotes, endnotes and comments, each line labelled (see Hidden content).
.pptxpptxText per slide in presentation order; the slide’s speaker notes follow a [Speaker notes] line in the same slide.
.odtodtParagraphs, headings, lists, tables and frames. Annotations are not indexed.
.odpodpText per slide, with speaker notes after a [Speaker notes] line.
.epubepubText per chapter, in spine order.
.doc (Word 97–2003)docThe body text only. Headers, footers, notes, comments and text boxes are not indexed.
.msg (Outlook)msgSubject, From, To, Cc, Bcc and Date, the body, and the attachment names and sizes.
.emlemlThe same mail shape as .msg.
.rtfrtfThe text; hidden text is indexed without a label.

For mail (.msg, .eml), attachments — embedded messages included — are listed by name and size under an [Attachments] line and are never opened. Links and cid: references are never fetched. A .msg whose body is held only as compressed RTF has its headers and attachments indexed, not its body.

ExtensionParserNotes
.xlsxxlsxEvery sheet in workbook order, hidden sheets, rows and columns and cell comments included and labelled. Formulas are read as their cached values and date cells as ISO dates. In a user’s library, also an outline, a table-tool source and its pictures.
.odsodsEvery sheet, in the same shape as xlsx. Not a table-tool source.
.csv, .tsvdelimitedIndexed by line, each chunk with the header row’s column names. In a user’s library, also an outline and a table-tool source.
ExtensionsParser
.txt, .md, .log, .yaml, .ymltext
Code and config: .js, .ts, .py, .java, .c, .h, .cpp, .go, .rs, .rb, .sh, .sql, .toml, .ini, .csstext (stored as text/plain)
.jsonjson
.ndjson, .jsonlndjson
.xml, .svgxml (the text content)
.html, .htmhtml (the text; scripts, styles, comments and attributes are dropped)

Every text-kind file is indexed with line locations, so a search hit points at the lines it came from. A file with an extension DioscHub does not know, whose bytes are valid text, is also read as text.

KindExtensions
image.png, .jpg, .jpeg, .gif, .webp, .bmp, .tiff, .tif, .avif
media.mp3, .wav, .mp4, .m4a, .mov, .webm, .mkv, .ogg, .flac, .avi
archive.zip, .gz, .tar, .7z, .rar, .bz2, .xz

Only PNG, JPEG, WebP and GIF images are sent to a vision model; the other image formats are stored and listed but not given to the model.

  • .xls (Excel 97–2003) is recognised but stored as other (stored_only_kind): it is not indexed and is not a table-tool source. Save it as .xlsx to have it read.
  • .ppt, OCR of scanned PDFs and images, and audio transcription are not supported.
  • PDF white text is indexed as ordinary text; it is not detected or labelled as hidden.

Text a reader would not see on the page is still indexed, and each line of it starts with a label, so the model can tell it apart from visible content:

LabelWhere it comes from
[Hidden sheet] Every line of an xlsx sheet whose state is hidden or very hidden.
[Hidden row] An xlsx row marked hidden.
[Hidden column] <cell>: <value>A non-empty cell in a hidden xlsx column, after its row’s line.
[Comment] An xlsx cell comment ([Comment] B4: …), or a docx comment.
[Header] , [Footer] , [Footnote] , [Endnote] The matching docx parts, after the body.
[Hidden text] A docx run formatted as hidden, as its own line after its paragraph.

Labels combine outer first ([Header] [Hidden text] …). A line break, a ] or a fake label inside hidden content is flattened, so hidden text cannot make itself look visible. Hidden pptx slides are indexed like any other slide.

Classification runs once, on every path that stores a file — a widget upload, MCP put_file, and knowledge-base upload and sync — after the bytes are staged and before the file record is written. It reads the file’s first 8 KiB, its name and the type the uploader declared. None of the three is trusted alone:

  1. Executables are refused, whatever the name or declared type. A file is an executable when its bytes start like a Windows (PE), Linux (ELF) or macOS (Mach-O) binary, when its extension is .exe, .dll, .msi, .scr, .com, .sys, .so or .dylib, or when its declared type is an executable MIME type (application/x-msdownload, application/x-executable, application/vnd.microsoft.portable-executable and their aliases). An ELF binary renamed notes.txt and declared text/plain is refused.
  2. An admin rule on the extension decides next, if there is one.
  3. The claim. A known extension is the claim, and the declared type never overrides it. Only when the extension is unknown does a declared type count: an admin rule on that MIME type, else the built-in table. An empty type, application/octet-stream and text/plain are no claim (text/plain is what a multipart upload sends when the part has no type).
  4. The bytes must fit the claim. A PDF must start %PDF-, an Office Open XML or OpenDocument file must be a zip, a .doc or .msg a compound file, a text type valid text, an image its own signature, and so on. Bytes that don’t fit are stored as other with type_mismatch.
  5. No claim at all (no extension, or one nobody knows, and no usable declared type): a recognised signature decides — a PDF, PNG, JPEG, zip and so on — else valid text is text, else the file is other with stored_only_kind.

So application/octet-stream uploads and files with no extension are classified, not refused. A zip-based document with no extension is recognised only as a zip (archive), so keep extensions on uploads.

The file is stored with the classified MIME type, not the one the uploader declared; an other file is stored as application/octet-stream. When the file is served back:

  • an other file is always sent as application/octet-stream, as an attachment;
  • only PNG, JPEG, GIF, WebP, BMP and AVIF images, PDF and text/plain are ever served inline; everything else, HTML and SVG included, is an attachment.

Every content response carries X-Content-Type-Options: nosniff and a sandboxing Content Security Policy.

An executable is refused with HTTP 415 on every path, before any file record is written; the staged bytes are deleted:

{
"statusCode": 415,
"code": "FILE_TYPE_REFUSED",
"message": "This file type is not accepted"
}

A knowledge-base replace of an existing file with an executable leaves the existing file and its version as they were. The refusal is the only type-based refusal; every other file is stored, as other if nothing else fits.

A file type rule tells DioscHub how to treat a type the built-in table does not cover, such as HL7 v2 messages read as text, with a hint about what the file holds. Rules are deployment-wide system settings, not per Role, and apply to the next upload on every path.

System → File types in the admin portal. The page has three parts:

  • Saved rules — the rule list: Key type (Extension or MIME type), Key, Kind, Parser and Hint. Add, edit or remove rows, then Save. Saving replaces the whole list.
  • Test a file — pick a file and see how DioscHub would classify it with the saved rules (not unsaved edits on the page). Only the first 64 KiB are sent and nothing is stored. The result shows the outcome, kind, MIME type, parser, hint, stored-only reason and the matching rule, and for a text parser a preview of the first 2,000 characters.
  • Built-in types — the read-only built-in table.
PermissionGrantsPredefined roles
file-types:readSee the File types tab and its rulesViewer, Operator, Admin, Owner
file-types:manageSave rules and use Test a fileAdmin, Owner

Both are in the File types group when you build a custom admin role, and both are available on every tier. Saving the rules and running a test are recorded in the audit trail; the test’s record notes that it ran and does not hold the file.

A rule maps one exact key to a kind, a parser and an optional hint:

FieldRule
extension or mimeTypeExactly one of the two. An extension is matched exactly, without the dot, case-insensitive: 1–32 characters, letters and digits, then also _, + and - (a leading . is dropped). A MIME type is one exact type/subtype. No wildcards, globs or patterns.
kinddocument, text, table, media, archive or other.
parserOne of the parsers allowed for the kind (below).
hintOptional, at most 500 characters, no control characters. Saved with each file the rule matches.
KindParsers a rule may choose
texttext, json, ndjson, xml, html, none
tabledelimited, xlsx, ods, none
documentpdf, docx, pptx, odt, odp, epub, doc, msg, eml, rtf, none
media, archive, othernone

Parser none stores the file without reading it. A rule has no other fields: there are no parser options, no custom code, and no setting for inline serving or vision.

  • An extension rule is checked before the built-in table, so a rule on a non-image built-in extension (for example .log) replaces the built-in entry for that extension.
  • A MIME rule applies only when the file’s extension is not in the built-in table and has no rule, and the uploader declared exactly that type.
  • A matched rule’s file is stored with the parser’s standard MIME type (text/plain for text, application/json for json, text/csv for delimited, and so on; application/octet-stream for none).
  • The bytes must still fit the parser: a rule with parser docx on bytes that are not a zip gives other / type_mismatch, and a text parser needs valid text.
  • Executables are refused before any rule is looked at.

The hint is stored in the file’s metadata (typeHint) and is returned by Test a file. This release does not yet add it to what the model is told about a file.

PUT /api/admin/file-types/rules with { "rules": [ … ] } replaces the list. If any rule is invalid, nothing is saved and the answer is 400:

{
"code": "FILE_TYPE_RULES_INVALID",
"errors": [{ "index": 0, "field": "extension", "code": "executable_key" }]
}
CodeWhy
executable_keyThe key is an executable extension or MIME type. Executables can never be allowed.
image_keyThe key is a built-in image extension or any image/ MIME type. Rules can’t change what reaches vision.
bad_extension, bad_mimeThe key is malformed. text/plain and application/octet-stream are refused as MIME keys: they are no claim.
both_keys, no_keyA rule needs exactly one of extension and mimeType.
bad_kindNot one of the six rule kinds. image is not a rule kind.
bad_parserThe parser is not in the fixed set for the kind.
bad_hintOver 500 characters, or holds a control character.
unknown_fieldAny field outside the five above, such as a pattern, options, or an inline or vision flag.
duplicate_keyTwo rules with the same key.
too_many_rules, not_arrayMore than 200 rules, or rules is not a list.

The rules can also be read with GET /api/admin/file-types (file-types:read), which returns the built-in table, the saved rules, the kinds and parsers a rule may use and these limits. Test a file is POST /api/admin/file-types/test (file-types:manage, multipart file, at most 64 KiB). The rules are not part of GET or PUT /api/admin/system-settings; a files.typeRules value sent there is ignored.

A saved rule is checked again each time it is used, and one that no longer validates is skipped. In a cluster, saving the rules notifies the other instances, which reload them.

Every parser is bounded, so a hostile or broken file can’t exhaust the hub. A file that hits a limit ends failed with the reason parser_limit — “The file is too large to read.” It is kept, not indexed, and not retried. Each limit is an environment variable on the hub; a value that is not a positive number falls back to the default.

LimitDefaultVariable
Input size, pdf, docx, xlsx, pptx, odt, ods, odp, epub, doc, msg256 MiB eachFILE_INDEXING_<PARSER>_MAX_BYTES
Input size, text, json, delimited, ndjson512 MiB eachFILE_INDEXING_<PARSER>_MAX_BYTES
Input size, html, xml, eml, rtf64 MiB eachFILE_INDEXING_<PARSER>_MAX_BYTES
Zip entries10,000FILE_INDEXING_ZIP_MAX_ENTRIES
Zip total uncompressed size512 MiBFILE_INDEXING_ZIP_MAX_UNCOMPRESSED_BYTES
Zip compression ratio200FILE_INDEXING_ZIP_MAX_RATIO
Time to read one file300,000 ms (5 minutes)FILE_INDEXING_PARSE_MAX_MS
Image size (width × height)40,000,000 pixelsVISION_MAX_INPUT_PIXELS

<PARSER> is the parser name in capitals, for example FILE_INDEXING_PDF_MAX_BYTES.

  • Input size is checked against the stored size before any byte is read.
  • Zip limits apply to every zip-based format (docx, xlsx, pptx, odt, ods, odp, epub). The zip’s directory is checked before anything is inflated. The ratio applies to each entry over 1 MiB and to the whole archive once it is over 1 MiB. Zip64 archives and directories that don’t add up are refused, and an entry is never inflated past the size it declares.
  • Time counts from opening the file to its last chunk, indexing included; when it runs out the read is stopped and its worker freed. A worker that stops responding is a separate case and ends failed with the reason timeout.
  • Images are not parsed for text, so this limit does not fail a file. An image over it is never sent to a vision model: its size is read from the image header and checked before anything is decoded, and the image stays stored. The same limit caps the other images the hub decodes.
  • XML is never allowed to fetch an external entity or expand an entity declared in a DTD, in any format.
  • Fixed limits: mail nesting depth 64 and headers 1 MiB (.eml), RTF group depth 1,024, at most 1,000 attachment lines per message, and bounded row and cell repetition in .ods.
  • Legacy Office files (.doc, .msg) have their compound-file structure checked before the parser reads them, and are read in an isolated process with its own memory cap; running out of memory there ends parser_limit and the hub carries on.

These limits apply on every tier.

The table tools (describe_workbook, load_table, query_table) work on .xlsx, CSV and TSV files in a user’s library. Their limits are fixed in code, the same on every tier, and none is an environment variable. Going past one gives the model a fixed sentence, never an error.

LimitValue
Source file load_table reads256 MB (268,435,456 bytes), checked against the stored size before the file is read
One loaded table1,000,000 rows, 512 columns, 20,000,000 cells; a CSV at most 1,000,000 records
Header rows (headerRows)0 to 10, default 1
Loaded tables kept per file8; loading a ninth drops the one used least recently
A table that isn’t kept because the storage quota is fullUsable on that hub instance for 1 hour
describe_workbook answer24,000 characters; later sheets by name and visibility only
Table regions listed per sheet20
load_table answer16,000 characters; 5 sample rows of up to 30 columns, values cut to 80 characters
query_table answer1,000 rows and 16,000 characters; values cut to 200 characters
query_table run time10 seconds
Pictures extracted per workbook20, each at most 16 MiB in the workbook and scaled to at most 1,568 pixels on the longer side and 4.5 MiB; 32 MiB of scaled pictures per workbook

The stored outline itself is at most 2 MiB. Sheets after the 100th carry only their name, visibility and used range, and when the outline would still be larger, previews are left out first, then regions, then the other details, each with a note in the outline.

Each loaded table is built in a fresh, locked-down database with no file or network access and no extensions, using at most 1,024 MiB of memory and 2 threads.

A rule change applies to new uploads only; files already stored keep the kind they were given. To apply a new rule to existing files, re-index them: a re-index classifies the stored file again from its stored bytes, its name and its stored MIME type with the current rules, then reads it by the new kind. It rewrites the file’s kind, MIME type and hint.

On System → Files → Bulk Re-index the filter has Kind, Extension and MIME type alongside scope, status, reason and embedding model. The filter matches each file’s current kind and type. For example, after adding an hl7 rule, filter on Kind other and Extension hl7, Estimate, then confirm. The estimate needs settings:read and starting a re-index settings:manage.

  • An extension rule reaches every stored file with that extension.
  • A MIME rule reaches only files stored with that MIME type. A file that was stored as other has the type application/octet-stream, so a MIME rule added later does not reach it.
  • A stored file that turns out to be an executable is not deleted; it becomes other / type_mismatch and is not read.
  • Derived and generated files are not re-classified.

The upgrade that adds kinds gives every existing file a kind from its stored MIME type; it can’t read the bytes. A file stored under a MIME type the built-in table doesn’t list (for example a code file a browser sent as text/x-python) becomes other, and is then served as a download and not read. Re-index those files — Kind other — to classify them from their bytes.

The widget’s file picker offers the types the hub lists in acceptedTypes from GET /api/files/policy: a . extension for every built-in type and every extension rule, then the MIME type of every MIME rule. The list is the same for every user, Role and anonymous visitor, and a rule change shows in the next policy read.

It is a hint for the picker, not a gate. Drag and drop, paste and files from client-side tools are not filtered by it, and the hub accepts and classifies any file that is not an executable. To offer a different list, set the accept property (accept attribute) of the <diosc-file-upload> element; when set, even to an empty string, it replaces the hub’s list.

Classification, the formats, the parser limits and the File types page are not tier-gated. They apply wherever files are stored; users can upload chat files only where their Role has the files feature (see Licensing & tiers). Moving rules to another environment with configuration transfer needs the Enterprise tier.


Related: File storage limits, quota & retention · System settings · Configuration transfer · Knowledge bases