Compare list/dict; parse JSON/CSV at scale
Python Data Structures Comparison
Precisely compare Python's list and dict data structures in CPython 3. For the operations append, insert, lookup, update, and delete, state their average and worst-case time complexity, discuss their memory implications, and describe their ordering guarantees. Provide concise Python code examples for appending an item to a list and updating a dictionary.
Data Formats: JSON vs. CSV
Define JSON and CSV data formats. Discuss scenarios where you would choose JSON over CSV, considering factors such as data nesting, schema evolution, interoperability, and compression characteristics.
Stream Processing Large Files
Given a 10 GB data file and a machine with 16 GB of RAM, demonstrate how to efficiently process large data files without exceeding memory limits. Your task is to compute the sum of a numeric column named 'value' from a file named data.csv.
Provide exact Python code for stream-reading and processing data using the following methods:
- JSON Lines: Iterate line-by-line and parse each line using
json.loads. - CSV: Utilize Python's
csv.DictReader. - Pandas: Employ
pandas.read_csvwith thechunksizeparameter.
Additionally, explain your approach to handling common data quality issues during this process, including:
- Malformed rows
- Missing or NaN values
- Bad encodings
- Numeric overflow
Propose practical chunk-size heuristics for processing a 10 GB file on a 16 GB RAM machine. Finally, suggest a non-pandas alternative that still safely streams and processes the data without loading the entire file into memory.
AI interview prep
Core dilemma
Balancing efficient in-memory processing of large datasets with robust error handling and resource management under strict memory constraints.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compare list/dict; parse JSON/CSV at scale
Python Data Structures Comparison
Precisely compare Python's list and dict data structures in CPython 3. For the operations append, insert, lookup, update, and delete, state their average and worst-case time complexity, discuss their memory implications, and describe their ordering guarantees. Provide concise Python code examples for appending an item to a list and updating a dictionary.
Data Formats: JSON vs. CSV
Define JSON and CSV data formats. Discuss scenarios where you would choose JSON over CSV, considering factors such as data nesting, schema evolution, interoperability, and compression characteristics.
Stream Processing Large Files
Given a 10 GB data file and a machine with 16 GB of RAM, demonstrate how to efficiently process large data files without exceeding memory limits. Your task is to compute the sum of a numeric column named 'value' from a file named data.csv.
Provide exact Python code for stream-reading and processing data using the following methods:
- JSON Lines: Iterate line-by-line and parse each line using
json.loads. - CSV: Utilize Python's
csv.DictReader. - Pandas: Employ
pandas.read_csvwith thechunksizeparameter.
Additionally, explain your approach to handling common data quality issues during this process, including:
- Malformed rows
- Missing or NaN values
- Bad encodings
- Numeric overflow
Propose practical chunk-size heuristics for processing a 10 GB file on a 16 GB RAM machine. Finally, suggest a non-pandas alternative that still safely streams and processes the data without loading the entire file into memory.