Big Data Encodings
These encodings are often used with HDFS or some other distributed file system. Since the data can be as large as terabytes or petabytes, it is crucial to encode files in a space optimal way and also allow themselves to be read or written in an optimal way. For this reason, these encodings are generally not human readable (see the next section for more details on these).
| File Encoding | Storage Format | Compression | Intended for | Schema Evolution | Other Notes |
|---|---|---|---|---|---|
| Avro | Row | Weak | Writes | Great | Works well with Kafka. Similar to Google Protobuf and Apache Thrift; allows for versioning and provides backwards/forward compatibility with schemas. |
| Parquet | Columnar | Good | Reads | Limited | Works well with Spark. Can store highly nested data. |
| ORC | Columnar | Great | Reads | Good | Works well with Hive. Supports ACID transactions. |
Human Readable Encodings
These encodings are often used to structure data in a more convenient way for developers and end-users, with less importance on the data size (in memory or disk). For this reason, these formats are typically human readable.
For example, CSV and TSV are very popular output formats for data analysts who may use programs like Microsoft Excel. JSON formats are also very popular for passing data around with REST APIs, and it is convenient for web browsers since it works natively with Javascript.
| File Encoding | Characteristics | Caveats | Other Notes |
|---|---|---|---|
| CSV, TSV, PSV | Comma Separated Values Tab Separated Values Pipe Separated Values |
Cannot distinguish between string types and number types Ambiguity with parsing delimited values and newlines across implementations Ambiguity with column headers that might or might not be present Some formats are unescaped, causing additional complications when rows have embedded values such as double-quotes. |
Lightweight, simple, widely used |
| JSON | Can represent deeply nested data Key-value dictionary is easy to read Works well for representing data on the front-end (Javascript) |
Has issues representing numbers with precision (i.e. larger than 2^52) Can be quite bloaty in size Cannot insert raw binary strings; a hacky workaround is to encode them with Base64 |
Simple, widely used with REST/GraphQL |
| Binary JSON (e.g. BSON) | Represents JSON as a binary sequence to save space | End result is not human-readable. Space savings compared to original JSON are limited. |
|
| XML | Schema-friendly Verbose Can represent nested data |
Similar to JSON, has issues representing numbers with precision (i.e. larger than 2^52) and inserting raw binary strings. Similar to CSV, it cannot distinguish between string types and number types Size is bloaty, and XML template can be too verbose and hard to read (as a human) |
Used widely for configuration files and SOAP architectures. |