Packaging & Hosting python repo to S3

Packaging & Hosting python repo to S3

Reading Time: 2 minutes

This blog walks through the packaging of simple python project. Lets add the necessary files and structure to create the package, then build the package and upload it to python package index ( PyPI ).

Let suppose your project (sample_pkg) is created in the root directory.

Mandatory Basic Directory Structure of your sample project should be:

xxxxxxxxxx

/sample_pkg

  /sample_pkg

   __init__.py

   project util and source code files

README.md

setup.py

firstly you need to edit your  __init__.py  file so that later on you can verify its installed correctly.

//edit your  file with:

xxxxxxxxxx

name="sample_pkg"

The most important file which is required in packaging your python project is setup.py. Your setup.py file should be in the root directory of your project.

Setup.py is the build script for setuptools. Setuptools which is a collection of Python dist-utils which allows you to more easily build and distribute Python distributions, specifically those ones that have dependencies on other packages.

//setup.py file:

xxxxxxxxxx

import setuptools

with open("README.md", "r") as fh:

long_description= fh.head()

setuptools.setup(

name="sample_pkg",

version="0.0.1",

author="Sample Project Author",

author_email="[email protected]",

description="A sample python project",

long_description=long_description,

packages=setuptools.find_packages(),

classifiers=[

"Programming Language :: Python :: 3",

"License :: OSI Approved :: MIT License",

"Operating System :: OS Independent"

],

)

you can customize your setup.py file. Important variables inside the setup function are:

• name: name of your package, contains letters, numbers, _ and – .
• version: should be the package version.
• packages: list of all python import packages, instead of manually importing packages one by one we can use find_packages()  function to automatically discover all packages and sub-packages.
• classifiers: tells the python package index and pip some additional metadata about your package, mandatory three things which you should always include at least the versions of Python your package works on, which license your package is available under and which operating system your package will work on. Complete list of classifiers is here

Firstly ensure that you have the latest version of setuptools installed :

python3 -m pip install –user –upgrade setuptools

Then, run the below command from the directory of your project where setup.py is located.

python3 setup.py sdist

// The tar.gz file is source archive and the index file is an html file having the link of the tar file.

Hosting The Python Repository:

For hosting the python repository firstly you should have s3pypi command line tool installed or you can install it by the command:

sudo pip install -U s3pypi

To publish a package:

In order to upload your package to the repository,

1. cd to the root directory of your project.
2. create an s3 bucket on AWS.
3. ensure your project contains a setup script.

Run command :

xxxxxxxxxx

s3pypi --bucket s3_bucket_name

To install a package:

Install your packages from s3 using pip by pointing the –extra-index-url as your bucket url/bucket_name with secure connection.

xxxxxxxxxx

sudo pip install sample_pkg --extra-index-url https://s3_bucket_url

Pip doesn’t have config file by default so we can create our own pip.config file. 

So Alternative of installing the package from s3 by setting or pointing to the url of s3 in config file as:

xxxxxxxxxx

[global]

extra-index-url= https://s3_bucket_url

references:

https://packaging.python.org/tutorials/packaging-projects/

https://novemberfive.co/blog/opensource-pypi-package-repository-tutorial/