File types, kinds & parser limits
Every file DioscHub stores is given a kind when it arrives. The kind decides what happens next: whether the file is read and indexed, which parser reads it, whether it can reach the model as an image, and how it is served back. One built-in table drives all of it, and administrators can extend it with file type rules on System → File types.
This page covers the kinds and formats, how classification works, the rules, the limits that bound every parser, and how to re-classify files that are already stored. The upload size limit, storage quota and retention are on File storage limits, quota & retention.
| Kind | What happens to the file |
|---|---|
document | Read by its parser and indexed for search. |
table | Read and indexed. XLSX, CSV and TSV files in a user’s library also get a workbook outline and can be queried with the table tools. |
text | Read as text and indexed, each chunk located by line numbers. Code, markup and data files are text. |
image | Stored and marked ready; no text is extracted. PNG, JPEG, WebP and GIF images can be sent to a vision model. |
media | Audio and video. Stored only, not read. |
archive | Zip, gzip, tar and the like. Stored only, not opened. |
other | Anything DioscHub does not recognise, or whose bytes don’t match their claimed type. Stored only, and always served as a download. |
A file that is stored only has the status stored_only with one of two reasons:
stored_only_kind— “This kind of file is stored but not read.” (media, archives and unknown files);type_mismatch— “The file’s content doesn’t match its type.”
Files DioscHub derives for its own use have a fixed kind: derived for a cached table, and image for a
picture extracted from a workbook. Neither is ever listed as one of the user’s files. Files a tool
generates (generate_file, bundle_files) get the kind of the format the generator produced; rules are
not applied to them.
Supported formats
Section titled “Supported formats”The built-in table. System → File types shows the same list under Built-in types.
Document
Section titled “Document”| Extension | Parser | What is indexed |
|---|---|---|
.pdf | pdf | The text layer, by page. |
.docx | docx | Body text, then headers, footers, footnotes, endnotes and comments, each line labelled (see Hidden content). |
.pptx | pptx | Text per slide in presentation order; the slide’s speaker notes follow a [Speaker notes] line in the same slide. |
.odt | odt | Paragraphs, headings, lists, tables and frames. Annotations are not indexed. |
.odp | odp | Text per slide, with speaker notes after a [Speaker notes] line. |
.epub | epub | Text per chapter, in spine order. |
.doc (Word 97–2003) | doc | The body text only. Headers, footers, notes, comments and text boxes are not indexed. |
.msg (Outlook) | msg | Subject, From, To, Cc, Bcc and Date, the body, and the attachment names and sizes. |
.eml | eml | The same mail shape as .msg. |
.rtf | rtf | The text; hidden text is indexed without a label. |
For mail (.msg, .eml), attachments — embedded messages included — are listed by name and size under an
[Attachments] line and are never opened. Links and cid: references are never fetched. A .msg whose
body is held only as compressed RTF has its headers and attachments indexed, not its body.
| Extension | Parser | Notes |
|---|---|---|
.xlsx | xlsx | Every sheet in workbook order, hidden sheets, rows and columns and cell comments included and labelled. Formulas are read as their cached values and date cells as ISO dates. In a user’s library, also an outline, a table-tool source and its pictures. |
.ods | ods | Every sheet, in the same shape as xlsx. Not a table-tool source. |
.csv, .tsv | delimited | Indexed by line, each chunk with the header row’s column names. In a user’s library, also an outline and a table-tool source. |
| Extensions | Parser |
|---|---|
.txt, .md, .log, .yaml, .yml | text |
Code and config: .js, .ts, .py, .java, .c, .h, .cpp, .go, .rs, .rb, .sh, .sql, .toml, .ini, .css | text (stored as text/plain) |
.json | json |
.ndjson, .jsonl | ndjson |
.xml, .svg | xml (the text content) |
.html, .htm | html (the text; scripts, styles, comments and attributes are dropped) |
Every text-kind file is indexed with line locations, so a search hit points at the lines it came from. A file
with an extension DioscHub does not know, whose bytes are valid text, is also read as text.
Image, media and archive
Section titled “Image, media and archive”| Kind | Extensions |
|---|---|
image | .png, .jpg, .jpeg, .gif, .webp, .bmp, .tiff, .tif, .avif |
media | .mp3, .wav, .mp4, .m4a, .mov, .webm, .mkv, .ogg, .flac, .avi |
archive | .zip, .gz, .tar, .7z, .rar, .bz2, .xz |
Only PNG, JPEG, WebP and GIF images are sent to a vision model; the other image formats are stored and listed but not given to the model.
Not read yet
Section titled “Not read yet”.xls(Excel 97–2003) is recognised but stored asother(stored_only_kind): it is not indexed and is not a table-tool source. Save it as.xlsxto have it read..ppt, OCR of scanned PDFs and images, and audio transcription are not supported.- PDF white text is indexed as ordinary text; it is not detected or labelled as hidden.
Hidden content
Section titled “Hidden content”Text a reader would not see on the page is still indexed, and each line of it starts with a label, so the model can tell it apart from visible content:
| Label | Where it comes from |
|---|---|
[Hidden sheet] | Every line of an xlsx sheet whose state is hidden or very hidden. |
[Hidden row] | An xlsx row marked hidden. |
[Hidden column] <cell>: <value> | A non-empty cell in a hidden xlsx column, after its row’s line. |
[Comment] | An xlsx cell comment ([Comment] B4: …), or a docx comment. |
[Header] , [Footer] , [Footnote] , [Endnote] | The matching docx parts, after the body. |
[Hidden text] | A docx run formatted as hidden, as its own line after its paragraph. |
Labels combine outer first ([Header] [Hidden text] …). A line break, a ] or a fake label inside hidden
content is flattened, so hidden text cannot make itself look visible. Hidden pptx slides are indexed like any
other slide.
How a file is classified
Section titled “How a file is classified”Classification runs once, on every path that stores a file — a widget upload, MCP put_file, and
knowledge-base upload and sync — after the bytes are staged and before the file record is written. It reads
the file’s first 8 KiB, its name and the type the uploader declared. None of the three is trusted alone:
- Executables are refused, whatever the name or declared type. A file is an executable when its bytes
start like a Windows (PE), Linux (ELF) or macOS (Mach-O) binary, when its extension is
.exe,.dll,.msi,.scr,.com,.sys,.soor.dylib, or when its declared type is an executable MIME type (application/x-msdownload,application/x-executable,application/vnd.microsoft.portable-executableand their aliases). An ELF binary renamednotes.txtand declaredtext/plainis refused. - An admin rule on the extension decides next, if there is one.
- The claim. A known extension is the claim, and the declared type never overrides it. Only when the
extension is unknown does a declared type count: an admin rule on that MIME type, else the built-in
table. An empty type,
application/octet-streamandtext/plainare no claim (text/plainis what a multipart upload sends when the part has no type). - The bytes must fit the claim. A PDF must start
%PDF-, an Office Open XML or OpenDocument file must be a zip, a.docor.msga compound file, a text type valid text, an image its own signature, and so on. Bytes that don’t fit are stored asotherwithtype_mismatch. - No claim at all (no extension, or one nobody knows, and no usable declared type): a recognised
signature decides — a PDF, PNG, JPEG, zip and so on — else valid text is
text, else the file isotherwithstored_only_kind.
So application/octet-stream uploads and files with no extension are classified, not refused. A zip-based
document with no extension is recognised only as a zip (archive), so keep extensions on uploads.
What is stored and served
Section titled “What is stored and served”The file is stored with the classified MIME type, not the one the uploader declared; an other file is
stored as application/octet-stream. When the file is served back:
- an
otherfile is always sent asapplication/octet-stream, as an attachment; - only PNG, JPEG, GIF, WebP, BMP and AVIF images, PDF and
text/plainare ever served inline; everything else, HTML and SVG included, is an attachment.
Every content response carries X-Content-Type-Options: nosniff and a sandboxing Content Security Policy.
The executable refusal
Section titled “The executable refusal”An executable is refused with HTTP 415 on every path, before any file record is written; the staged
bytes are deleted:
{ "statusCode": 415, "code": "FILE_TYPE_REFUSED", "message": "This file type is not accepted"}A knowledge-base replace of an existing file with an executable leaves the existing file and its version
as they were. The refusal is the only type-based refusal; every other file is stored, as other if
nothing else fits.
File type rules
Section titled “File type rules”A file type rule tells DioscHub how to treat a type the built-in table does not cover, such as HL7 v2 messages read as text, with a hint about what the file holds. Rules are deployment-wide system settings, not per Role, and apply to the next upload on every path.
Where they live
Section titled “Where they live”System → File types in the admin portal. The page has three parts:
- Saved rules — the rule list: Key type (Extension or MIME type), Key, Kind, Parser and Hint. Add, edit or remove rows, then Save. Saving replaces the whole list.
- Test a file — pick a file and see how DioscHub would classify it with the saved rules (not unsaved edits on the page). Only the first 64 KiB are sent and nothing is stored. The result shows the outcome, kind, MIME type, parser, hint, stored-only reason and the matching rule, and for a text parser a preview of the first 2,000 characters.
- Built-in types — the read-only built-in table.
| Permission | Grants | Predefined roles |
|---|---|---|
file-types:read | See the File types tab and its rules | Viewer, Operator, Admin, Owner |
file-types:manage | Save rules and use Test a file | Admin, Owner |
Both are in the File types group when you build a custom admin role, and both are available on every tier. Saving the rules and running a test are recorded in the audit trail; the test’s record notes that it ran and does not hold the file.
What a rule is
Section titled “What a rule is”A rule maps one exact key to a kind, a parser and an optional hint:
| Field | Rule |
|---|---|
extension or mimeType | Exactly one of the two. An extension is matched exactly, without the dot, case-insensitive: 1–32 characters, letters and digits, then also _, + and - (a leading . is dropped). A MIME type is one exact type/subtype. No wildcards, globs or patterns. |
kind | document, text, table, media, archive or other. |
parser | One of the parsers allowed for the kind (below). |
hint | Optional, at most 500 characters, no control characters. Saved with each file the rule matches. |
| Kind | Parsers a rule may choose |
|---|---|
text | text, json, ndjson, xml, html, none |
table | delimited, xlsx, ods, none |
document | pdf, docx, pptx, odt, odp, epub, doc, msg, eml, rtf, none |
media, archive, other | none |
Parser none stores the file without reading it. A rule has no other fields: there are no parser
options, no custom code, and no setting for inline serving or vision.
How rules apply
Section titled “How rules apply”- An extension rule is checked before the built-in table, so a rule on a non-image built-in extension
(for example
.log) replaces the built-in entry for that extension. - A MIME rule applies only when the file’s extension is not in the built-in table and has no rule, and the uploader declared exactly that type.
- A matched rule’s file is stored with the parser’s standard MIME type (
text/plainfortext,application/jsonforjson,text/csvfordelimited, and so on;application/octet-streamfornone). - The bytes must still fit the parser: a rule with parser
docxon bytes that are not a zip givesother/type_mismatch, and a text parser needs valid text. - Executables are refused before any rule is looked at.
The hint is stored in the file’s metadata (typeHint) and is returned by Test a file. This release does
not yet add it to what the model is told about a file.
What is rejected
Section titled “What is rejected”PUT /api/admin/file-types/rules with { "rules": [ … ] } replaces the list. If any rule is invalid,
nothing is saved and the answer is 400:
{ "code": "FILE_TYPE_RULES_INVALID", "errors": [{ "index": 0, "field": "extension", "code": "executable_key" }]}| Code | Why |
|---|---|
executable_key | The key is an executable extension or MIME type. Executables can never be allowed. |
image_key | The key is a built-in image extension or any image/ MIME type. Rules can’t change what reaches vision. |
bad_extension, bad_mime | The key is malformed. text/plain and application/octet-stream are refused as MIME keys: they are no claim. |
both_keys, no_key | A rule needs exactly one of extension and mimeType. |
bad_kind | Not one of the six rule kinds. image is not a rule kind. |
bad_parser | The parser is not in the fixed set for the kind. |
bad_hint | Over 500 characters, or holds a control character. |
unknown_field | Any field outside the five above, such as a pattern, options, or an inline or vision flag. |
duplicate_key | Two rules with the same key. |
too_many_rules, not_array | More than 200 rules, or rules is not a list. |
The rules can also be read with GET /api/admin/file-types (file-types:read), which returns the built-in
table, the saved rules, the kinds and parsers a rule may use and these limits. Test a file is
POST /api/admin/file-types/test (file-types:manage, multipart file, at most 64 KiB). The rules are not
part of GET or PUT /api/admin/system-settings; a files.typeRules value sent there is ignored.
A saved rule is checked again each time it is used, and one that no longer validates is skipped. In a cluster, saving the rules notifies the other instances, which reload them.
Parser limits
Section titled “Parser limits”Every parser is bounded, so a hostile or broken file can’t exhaust the hub. A file that hits a limit ends
failed with the reason parser_limit — “The file is too large to read.” It is kept, not indexed, and
not retried. Each limit is an environment variable on the hub; a value that is not a positive number falls
back to the default.
| Limit | Default | Variable |
|---|---|---|
Input size, pdf, docx, xlsx, pptx, odt, ods, odp, epub, doc, msg | 256 MiB each | FILE_INDEXING_<PARSER>_MAX_BYTES |
Input size, text, json, delimited, ndjson | 512 MiB each | FILE_INDEXING_<PARSER>_MAX_BYTES |
Input size, html, xml, eml, rtf | 64 MiB each | FILE_INDEXING_<PARSER>_MAX_BYTES |
| Zip entries | 10,000 | FILE_INDEXING_ZIP_MAX_ENTRIES |
| Zip total uncompressed size | 512 MiB | FILE_INDEXING_ZIP_MAX_UNCOMPRESSED_BYTES |
| Zip compression ratio | 200 | FILE_INDEXING_ZIP_MAX_RATIO |
| Time to read one file | 300,000 ms (5 minutes) | FILE_INDEXING_PARSE_MAX_MS |
| Image size (width × height) | 40,000,000 pixels | VISION_MAX_INPUT_PIXELS |
<PARSER> is the parser name in capitals, for example FILE_INDEXING_PDF_MAX_BYTES.
- Input size is checked against the stored size before any byte is read.
- Zip limits apply to every zip-based format (docx, xlsx, pptx, odt, ods, odp, epub). The zip’s directory is checked before anything is inflated. The ratio applies to each entry over 1 MiB and to the whole archive once it is over 1 MiB. Zip64 archives and directories that don’t add up are refused, and an entry is never inflated past the size it declares.
- Time counts from opening the file to its last chunk, indexing included; when it runs out the read is
stopped and its worker freed. A worker that stops responding is a separate case and ends
failedwith the reasontimeout. - Images are not parsed for text, so this limit does not fail a file. An image over it is never sent to a vision model: its size is read from the image header and checked before anything is decoded, and the image stays stored. The same limit caps the other images the hub decodes.
- XML is never allowed to fetch an external entity or expand an entity declared in a DTD, in any format.
- Fixed limits: mail nesting depth 64 and headers 1 MiB (
.eml), RTF group depth 1,024, at most 1,000 attachment lines per message, and bounded row and cell repetition in.ods. - Legacy Office files (
.doc,.msg) have their compound-file structure checked before the parser reads them, and are read in an isolated process with its own memory cap; running out of memory there endsparser_limitand the hub carries on.
These limits apply on every tier.
Table tool limits
Section titled “Table tool limits”The table tools (describe_workbook, load_table, query_table) work
on .xlsx, CSV and TSV files in a user’s library. Their limits are fixed in code, the same on every tier,
and none is an environment variable. Going past one gives the model a fixed sentence, never an error.
| Limit | Value |
|---|---|
Source file load_table reads | 256 MB (268,435,456 bytes), checked against the stored size before the file is read |
| One loaded table | 1,000,000 rows, 512 columns, 20,000,000 cells; a CSV at most 1,000,000 records |
Header rows (headerRows) | 0 to 10, default 1 |
| Loaded tables kept per file | 8; loading a ninth drops the one used least recently |
| A table that isn’t kept because the storage quota is full | Usable on that hub instance for 1 hour |
describe_workbook answer | 24,000 characters; later sheets by name and visibility only |
| Table regions listed per sheet | 20 |
load_table answer | 16,000 characters; 5 sample rows of up to 30 columns, values cut to 80 characters |
query_table answer | 1,000 rows and 16,000 characters; values cut to 200 characters |
query_table run time | 10 seconds |
| Pictures extracted per workbook | 20, each at most 16 MiB in the workbook and scaled to at most 1,568 pixels on the longer side and 4.5 MiB; 32 MiB of scaled pictures per workbook |
The stored outline itself is at most 2 MiB. Sheets after the 100th carry only their name, visibility and used range, and when the outline would still be larger, previews are left out first, then regions, then the other details, each with a note in the outline.
Each loaded table is built in a fresh, locked-down database with no file or network access and no extensions, using at most 1,024 MiB of memory and 2 threads.
Re-classify stored files
Section titled “Re-classify stored files”A rule change applies to new uploads only; files already stored keep the kind they were given. To apply a new rule to existing files, re-index them: a re-index classifies the stored file again from its stored bytes, its name and its stored MIME type with the current rules, then reads it by the new kind. It rewrites the file’s kind, MIME type and hint.
On System → Files → Bulk Re-index the filter has Kind, Extension and MIME type alongside
scope, status, reason and embedding model. The filter matches each file’s current kind and type. For
example, after adding an hl7 rule, filter on Kind other and Extension hl7, Estimate, then
confirm. The estimate needs settings:read and starting a re-index settings:manage.
- An extension rule reaches every stored file with that extension.
- A MIME rule reaches only files stored with that MIME type. A file that was stored as
otherhas the typeapplication/octet-stream, so a MIME rule added later does not reach it. - A stored file that turns out to be an executable is not deleted; it becomes
other/type_mismatchand is not read. - Derived and generated files are not re-classified.
After upgrading
Section titled “After upgrading”The upgrade that adds kinds gives every existing file a kind from its stored MIME type; it can’t read the
bytes. A file stored under a MIME type the built-in table doesn’t list (for example a code file a browser
sent as text/x-python) becomes other, and is then served as a download and not read. Re-index those
files — Kind other — to classify them from their bytes.
The widget’s file picker
Section titled “The widget’s file picker”The widget’s file picker offers the types the hub lists in acceptedTypes from
GET /api/files/policy: a . extension for every
built-in type and every extension rule, then the MIME type of every MIME rule. The list is the same for
every user, Role and anonymous visitor, and a rule change shows in the next policy read.
It is a hint for the picker, not a gate. Drag and drop, paste and files from client-side tools are not
filtered by it, and the hub accepts and classifies any file that is not an executable. To offer a
different list, set the accept property (accept attribute) of the <diosc-file-upload> element; when set,
even to an empty string, it replaces the hub’s list.
Classification, the formats, the parser limits and the File types page are not tier-gated. They apply
wherever files are stored; users can upload chat files only where their Role has the files feature (see
Licensing & tiers). Moving rules to another environment with
configuration transfer needs the Enterprise tier.
Related: File storage limits, quota & retention · System settings · Configuration transfer · Knowledge bases