Python – Writing large ZIP archives without memory inflation
source link: https://github.com/BuzonIO/zipfly
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Buzon - ZipFly
ZipFly is a zip archive generator based on zipfile.py. It was created by Buzon.io to generate very large ZIP archives for immediate sending out to clients, or for writing large ZIP archives without memory inflation.
Requirements
Python 3.5+
Install
pip3 install zipfly
Basic usage, compress on-the-fly during writes
Basic use case is compressing on the fly. Some data will be buffered by the zipfile deflater, but memory inflation is going to be very constrained. Data will be written to destination at fairly regular intervals.
ZipFly
default attributes:
- paths: [ ]
- mode: w
- chunksize: (bytes) 16384
- compression: Stored
- allowZip64: True
- compresslevel: None
- storesize: (bytes) 0
paths
list of dictionaries:
-
fs
(filesystem):path from your disk
-
n
(name):final path in zip file
import zipfly paths = [ { 'fs': 'home/user/Videos/jupiter.mp4', 'n': 'movies/jupiter.mp4', }, { 'fs': 'home/user/Documents/mercury.mp4', 'n': 'movies/mercury.mp4', }, ] zfly = zipfly.ZipFly( paths = paths ) generator = zfly.generator() print ( generator ) # <generator object generator at 0x7f85aad60b13> with open("test.zip", "wb") as f: for i in generator: f.write(i)
Examples
Create a ZIP file with size estimation
Use the BufferPredictionSize
to compute the correct size of the resulting archive.
Streaming a large file Efficient way to read a single very large binary file in python
Streaming multiple files in a zip The easiest is to use the Django or Flask built-in streaming feature
Maintainer
Santiago Debus (@santiagodebus.com)
Santiago's open-source projects are supported by his Patreon. If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.
License
This library was created by Buzon.io and is released under the MIT. Copyright 2019 Grow HQ, Inc.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK