Skip to content

Implement new library loading approach #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pombredanne merged 43 commits into from
May 31, 2021
Merged

Implement new library loading approach #20

pombredanne merged 43 commits into from
May 31, 2021

Conversation

@pombredanne
Copy link
Member

@pombredanne pombredanne commented May 27, 2021 •
edited
Loading

We now load the libmagic native library and its database from paths found in:

  1. environment variables,
  2. OR a location provider plugin,
  3. OR the system PATH,
  4. OR we fail with an informative error message.

Based on original code contributed by @priv-kweihmann to scancode-plugins
in aboutcode-org/scancode-plugins#9 and
moved here and adapted for use in the core code rather than in a plugin.

This also includes the merge of @tardyp #17 to search for common locations as a failover

Signed-off-by: Philippe Ombredanne [email protected]

Signed-off-by: Philippe Ombredanne [email protected]

pombredanne and others added 30 commits January 25, 2021 12:54
@pombredanne
Use wheels embedded in virtualenv.pyz
182532f 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Do not force an upgrade on virtualenv.pyz embeds
cd4e87b 
Signed-off-by: Philippe Ombredanne <[email protected]>
@steven-esser
Fix .gitattributes
51510cb 
Signed-off-by: Steven Esser <[email protected]>
@steven-esser
Merge pull request #21 from nexB/fix-gitattributes
386bb90 
Fix .gitattributes
@tardyp
failover to python-magic detection if vendored library not present
e7efce4 
This allows scancode to work on platforms not supported by typecode-libmagic
e.g. M1 Macs or windows WSL

Most of this code is from https://github.com/ahupp/python-magic/blob/e0ccc6d/magic/loader.py
by Adam Hupp, MIT License, compatible with typecode's Apache

Signed-off-by: Pierre Tardy <[email protected]>
@pombredanne
Merge pull request #24 from nexB/use-venv-embeds
1c84e9e 
Use virtualenv-embedded libraries
@pombredanne
Update markers syntax for pytest
d6fe59f 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add fallback version for setuptools_scm
ca6ab21 
This will work even from a git archive or when git is not installed.

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add note for setuptools_scam fallback version
1364bbb 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use azure-posix.yml for linux and macOS
be851b0 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Adopt new configure script derived from ScanCode
4f0aecf 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add notes on customization
aa04429 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Adopt new configure --dev convention
56ada8f 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Clarify CHANGELOG to be Rst
0dbcdc9 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add skeleton release notes to README.rst
d21aef3 
This was they do not end up in the template CHANGELOG.rst

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Implement new envt. variables approach
ae78d43 
We now load native libraries and executables from:
1. an envt. variable path
2. OR a locatin provider plugin
3. OR the PATH
or we fail

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Merge pull request #25 from nexB/new-configure
a75737d 
New configure
@pombredanne
Require latest commoncode
d77ff06 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Update CHANGELOG and README
7e9287a 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add note to vendor Pygments
a2f1535 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Merge latest skeleton
79388fb 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add vendy to testing/dev extras
9dfe6d4 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Bump vendored pygments to 2.9.0
f4d0960 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Make new libmagic loading work
50e1abb 
Its was skipping plugin loading. Also add tracing.

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Detect file type content only if no extension
0f5029a 
Only do content-based detection for programing language if there is
no file extension as this avoids a large number of detection issues.

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Adapt tests to latest Pygments
b480cf3 
Aslo ensure we use assertion in the correct order.

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use pip extras with the correct syntax
e5b373f 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Update CHANGELOG
5bbd8cb 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add CI links and format ReST
ca73e37 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Implement new library loading approach
001c831 
We now load the libmagic native library and its database from paths found in:
1. environment variables,
2. OR a location provider plugin,
3. OR the system PATH,
4. OR we fail with an informative error message.

Based on original code contributed by @priv-kweihmann to
scancode-plugins in aboutcode-org/scancode-plugins#9 and
moved here and adapted for use in the core code rather than in a plugin.

Contributed-by: Konrad Weihmann <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Merge branch 'failover' from @tardyp
1df5b73 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Rename test file
9963b44 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Remove unused import
b7f7337 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Format code and streamline failover support #17
68a2b7d 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Copy link
Member Author

pombredanne commented May 30, 2021

@tardyp I merged 17 in this branch which also carries over in a single place the code originally contributed by @priv-kweihmann for the typecode_libmagic_system_provided plugins aboutcode-org/scancode-plugins#9

Getting both of your reviews would be great before I merge

@pombredanne pombredanne mentioned this pull request May 30, 2021
Closed
@pombredanne pombredanne changed the title Implement new envt. variables approach Implement new library loding approach May 30, 2021
@pombredanne pombredanne changed the title Implement new library loding approach Implement new library loading approach May 30, 2021
tardyp
tardyp reviewed May 31, 2021
View reviewed changes
Copy link
Contributor

@tardyp tardyp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello,
Thanks for pinging me.

I am having hard time reviewing this 272 files PR.
Franckly I am not a huge fan of vendoring for opensource projects, although I do understand the need to limit dependency variablility for support reason.

I didn't file too much issue though, but I cannot claim a deep review. :-)

configure.bat Outdated
set PYTHON_EXECUTABLE=%PROVIDED_PYTHON%
goto run
if "%1" EQU "--python" (
echo "The --python is now DEPRECATED. Use the PYTHON_EXECUTABLE environment
Copy link
Contributor

@tardyp tardyp May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is an extra CR here.

Copy link
Member Author

@pombredanne pombredanne May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in https://github.com/nexB/skeleton and merged here

src/typecode/magic2.py
logger_debug('get_magicdb_location:', 'got path magicdb location:', magicdb_loc)

if not magicdb_loc:
warnings.warn(
Copy link
Contributor

@tardyp tardyp May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this not needed as libmagic will (probably) already complain if it does not find its db in standard location.

Copy link
Member Author

@pombredanne pombredanne May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point... I like keeping the warning on our side too though as the message would be more explicit

@pombredanne
Update format
3aeb2ec 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add Python 3.9 to Travis
2c412e8 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Format and remove spurious spaces
69eec23 
From #20

Reported-by: Pierre Tardy <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Merge latest skeleton
475c6ac 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Copy link
Member Author

pombredanne commented May 31, 2021

@tardyp you wrote:

I am having hard time reviewing this 272 files PR.
Franckly I am not a huge fan of vendoring for opensource projects, although I do understand the need to limit dependency variablility for support reason

I agree yet the Pygments vendoring here is a must here as we have dependencies on specific versions and vendoring is the sane approach to avoid conflicts (that popped up often in the past) when this fairly common Pygments library is installed otherwise.

@pombredanne
Copy link
Member Author

pombredanne commented May 31, 2021

@tardyp note also that we should be able to support Apple ARM chips as soon as I can find some CI supporting it.

tardyp
tardyp reviewed May 31, 2021
View reviewed changes
@pombredanne
Copy link
Member Author

pombredanne commented May 31, 2021

This looks decent now. Merging and releasing!

@pombredanne pombredanne merged commit d5a26bb into main May 31, 2021
@pombredanne pombredanne deleted the use-env-var branch May 31, 2021 17:38
JonoYang pushed a commit to aboutcode-org/purldb that referenced this pull request Apr 14, 2025
@pombredanne
Format and remove spurious spaces
94c382c 
From aboutcode-org/typecode#20

Reported-by: Pierre Tardy <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
JonoYang added a commit to aboutcode-org/purl-validator that referenced this pull request Oct 24, 2025
@JonoYang @pombredanne
@chinyeungli @TG1999 @keshav-space @AyanSinhaMahapatra @swastkk @arnav-mandal1234 @arijitde92 @OmkarPh @johnmhoran
Add initial code and update skeleton files (#26)
9bc3e43 
* Add skeleton release notes to README.rst

This was they do not end up in the template CHANGELOG.rst

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update format

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add Python 3.9 to Travis

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format and remove spurious spaces

From aboutcode-org/typecode#20

Reported-by: Pierre Tardy <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>

* Bump to more modern version of setuptools_scm

And remove v prefix from fallback version

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add space for correct syntax

Signed-off-by: Philippe Ombredanne <[email protected]>

* Create junction from Scripts to bin

 * This is handy for windows to have the same path as linux

Signed-off-by: Chin Yeung Li <[email protected]>

* Check for deps in local thirdparty directory #31

Signed-off-by: Jono Yang <[email protected]>

* Enforce use of requirements.txt #34

Signed-off-by: Jono Yang <[email protected]>

* Add scripts from scancode-toolkit/etc/release/ #33

Signed-off-by: Jono Yang <[email protected]>

* Normalize license in load_pkginfo_data #33

    * Create copyright statement from holder information

Signed-off-by: Jono Yang <[email protected]>

* Add --init option to configure #33

    * This is used for the case where we are starting off a project and have not yet generated requirements files

Signed-off-by: Jono Yang <[email protected]>

* Update README.rst #33

Signed-off-by: Jono Yang <[email protected]>

* Use venv as virtual environment directory name #37

    * Replace all references to `tmp` with `venv`

Signed-off-by: Jono Yang <[email protected]>

* Update configure.bat #33

    * Add --init option to configure.bat
    * Update help text in configure and configure.bat

Signed-off-by: Jono Yang <[email protected]>

* Add placeholder requirements.txt files #33

Signed-off-by: Jono Yang <[email protected]>

* Handle multiple options in configure #33

Signed-off-by: Jono Yang <[email protected]>

* Fix path to aboutcode in utils_thirdparty.py #33

    * Update README.rst

Signed-off-by: Jono Yang <[email protected]>

* Update release notes in README.rst

Signed-off-by: Jono Yang <[email protected]>

* Handle ExpressionParseError #33

    * Update README.rst with instructions for post-initialization usage

Signed-off-by: Jono Yang <[email protected]>

* Update README.rst

Signed-off-by: Philippe Ombredanne <[email protected]>

* Address review comments #33

    * Replace references to scancode-toolkit repo with links to the skeleton repo

    * Remove --python option from configure.bat

Signed-off-by: Jono Yang <[email protected]>

* Update READMEs

Signed-off-by: Jono Yang <[email protected]>

* Fixed #41 - Handled encoding issue when generating ABOUT files

Signed-off-by: Chin Yeung Li <[email protected]>

* Treat text files as text

And not a possible binaries
Also Ensure that we craft a minimally parsable license expression, even
if not correct.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add helper to publish files in GH releases

The upload is otherwise shaky.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add code to use curl if wget is not installed

Signed-off-by: Chin Yeung Li <[email protected]>

* Do not issue warning if thirdparty dir is missing

Signed-off-by: Philippe Ombredanne <[email protected]>

* Handle as_text correctly in cache

Signed-off-by: Philippe Ombredanne <[email protected]>

* Handle as_text correctly in cache

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add support for Python 3.10

Signed-off-by: Philippe Ombredanne <[email protected]>

* Adopt black style

Signed-off-by: Philippe Ombredanne <[email protected]>

* Drop Ubuntu 16 add Python 3.10

Signed-off-by: Philippe Ombredanne <[email protected]>

* Disable Python 3.10 tests on macOS 10.14

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add code to remove the symlink before creating

 * It will prompts "Cannot create a file when that file already exists."
if the symlink already exist.

Signed-off-by: Chin Yeung Li <[email protected]>

* Improve handling licenses without scancode

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add support for deb and rpm containers

Signed-off-by: Philippe Ombredanne <[email protected]>

* Support  licenses when ScanCode is not installed

Signed-off-by: Philippe Ombredanne <[email protected]>

* Improve wheel build

Allow to launch builds and then fetch built wheels later
Improve support for newer Pythons and OS versions.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Aligne with latest ScanCode TK updates

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not ignore setup.py

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update scripts

Make fetch_built_wheel work
Add new strip classifiers option to fix_thirdparty
Improve simple requirements parsing to get the latest versions

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not use pytest 7.0.0 which is buggy

Reference: pytest-dev/pytest#9608
Signed-off-by: Philippe Ombredanne <[email protected]>

* Update `configure` to work with space in installation path

Signed-off-by: Chin Yeung Li <[email protected]>

* Remove echo statement

Signed-off-by: Chin Yeung Li <[email protected]>

* automate pypi release on a tag

Signed-off-by: Tushar Goel <[email protected]>

* Refine GH Action definition

* Do not use explicit references to Python version and project name in
  descriptions
* Use Python 3.8 as a base
* Use only plain ASCII

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format GH action yaml

The indentations were not correct

Signed-off-by: Philippe Ombredanne <[email protected]>

* Use verbose name for job

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add black codestyle test for skeleton

- see aboutcode-org/skeleton#54

Signed-off-by: keshav-space <[email protected]>

* Update configure.bat

Signed-off-by: Chin Yeung Li <[email protected]>

* Deprecate windows-2016 images for azure CI

Modifies the azure CI for `vs2017-win2016` to `windows-2022` as 
the former will be deprecated very soon.

* Remove macos 10.14 job from azure-pipelines.yml

Signed-off-by: Jono Yang <[email protected]>

* Do not use Python 3.6 on Windows 2022 jobs

    * Python 3.6 is not available on Windows 2022 images

Signed-off-by: Jono Yang <[email protected]>

* Deprecate windows-2016 images for azure CI

* Modifies the azure CI for `vs2017-win2016` to `windows-2022` as the former will be deprecated very soon.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Remove macos 10.14 job from azure-pipelines.yml

Signed-off-by: Jono Yang <[email protected]>

* Do not use Python 3.6 on Windows 2022 jobs

    * Python 3.6 is not available on Windows 2022 images

Signed-off-by: Jono Yang <[email protected]>

* Run tests on macOS 11

Signed-off-by: Philippe Ombredanne <[email protected]>

* Align configuration scripts on POSIX and Windows

Doing so helps with keeping them in sync

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update README

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refactor and streamline thirdparty utilities

There is now a single primary script "fetch_thirdparty.py" that handles
everything in a simplified way. There is no utility to build wheels
anymore: these MUST be available before hand on PyPI or built
manually and uploaded in our self-hosted PyPI repository.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Cleanup whitespaces

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add usage instructions to README.rst

Signed-off-by: Jono Yang <[email protected]>

* Format code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Reorg setup sections

This is now organized with more important data first.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not depend on click. Use argparse.

These boostrap scripts cannot depend on click.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Correct configure scripts

These were no longer working

Signed-off-by: Philippe Ombredanne <[email protected]>

* Remove remnants of configure --init

This does not exists anymore

Signed-off-by: Philippe Ombredanne <[email protected]>

* Pytyon 3.6 is not available on Windows 2022

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add long_description_content_type

Twine and PyPI prefer having it.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Run fewer Azure jobs

This new configuration means that all the Python versions are tested
in a single CI job. This allows doing fewer checkouts and improves
CI throughput overall.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add RTD css templates

Adds changes to conf.py and html template theme_overrides.css
created by @johnmhoran

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add GitHub action for doc build tests

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Fix conf.py to fix doc build

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add docs option in configure

Adds a --docs option to the configure script to also install requirements
for the documentation builds.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add documentation contribute page

Adds documentation page on contributing to the docs, and
also modifies directory structure to avoid having the
skeleton directory in docs merged in projects.

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Properly check for existance of thirdparty dir

Signed-off-by: Jono Yang <[email protected]>

* Improve GH action documentation

Signed-off-by: Philippe Ombredanne <[email protected]>

* Use Python 3.9 as a base for actions

Signed-off-by: Philippe Ombredanne <[email protected]>

* Remove variable from string in fetch_thirdparty.py

    * The variable `environment` is not used when fetching sdists

Signed-off-by: Jono Yang <[email protected]>

* Improve thirdparty scripts

Ensure that site-package dir exists.
Other minor adjustments from a scancode-toolkit release

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update thirdparty fetching utilities

These were buggy in some corner cases.
They have been updated such that:
* --latest-version works.
* we can reliable fetch combinations of wheels and sdists for multiple
  OS combos at once
* we now support macOS universal wheels (for ARM CPUs)

Caching is now simpler: we have essentially a single file-based cache
under .cache. PyPI indexes are fetched and not cached, unless the new
--use-cached-index is used which can be useful when fetching many
thirdparty in a short timeframe.

The first PyPI repository in a list has precendence and we never fetch
from other repositories if we find wheels and sdsists there. This avoid
pounding too much on the self-hosted repo.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format code with black

Signed-off-by: Philippe Ombredanne <[email protected]>

* Only use PyPI for downloads

This is much faster

Signed-off-by: Philippe Ombredanne <[email protected]>

* Enable automatic release on tag

Signed-off-by: Philippe Ombredanne <[email protected]>

* Remove Travis config

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add ScanCode Code of Conduct

Signed-off-by: Philippe Ombredanne <[email protected]>

* Improve detection of thirdparty dir

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add comment

Signed-off-by: Philippe Ombredanne <[email protected]>

* Remove the thirdparty directory

Signed-off-by: Chin Yeung Li <[email protected]>

* Update the azure-piprlines

The following images are deprecated in GitHub actions and Azure DevOps:

* `ubuntu-18.04` : actions/runner-images#6002
* `macos-10.15` : actions/runner-images#5583

Due to this there was failing tests due to planned brownouts.

Updated ubuntu18 to ubuntu22
Updated macos-1015 to macos12

Signed-off-by: Chin Yeung Li <[email protected]>

* Add missing os import in utils_requirements.py

    * Rename references to etc/release with etc/scripts

Signed-off-by: Jono Yang <[email protected]>

* Reinstate Ubuntu 18 and drop Python 3.6 and 3.7

3.7 is not available anymore on newer OSes and is retired in 2023.

Signed-off-by: Philippe Ombredanne <[email protected]>

* Replace packaging with packvers

* import update in src/scripts/utils_dejacode

* Packvers replacing packaging in other src/scripts

* Added packvers in src/scripts/requirments.txt

Signed-off-by: swastik <[email protected]>

* Ignore egginfo

Signed-off-by: Philippe Ombredanne <[email protected]>

* Drop Python 3.7 add Python 3.11

Also test on latest Ubuntu and macOS

Signed-off-by: Philippe Ombredanne <[email protected]>

* Clean .cache and .eggs

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add COC to redistributed license-like files

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add new Makefile

Signed-off-by: Philippe Ombredanne <[email protected]>

* Align scripts with latest ScanCode Toolkit

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add pycodestyle in setup.cfg

Signed-off-by: Arnav Mandal <[email protected]>

* Remove deprecated github-actions/azure images

- remove deprecated `ubuntu-18.04` image
- remove deprecated `macos-10.15` image

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Pin Sphinx version to 6.2.1

Signed-off-by: Jono Yang <[email protected]>

* Publish PDF version of RTD documentation

Made changes in the .readthedocs.yaml to enable format for downloading pdf and epub versions of the documentation and added latex_elements in the conf.py file which generates the pdf without blank pages. The minimum version requirement for sphinx was 6.2.1 which was causing build failure in read the docs, hence changing it 3.3.1 as written in setup.cfg of nexB/aboutcode

Signed-off-by: Arijit De <[email protected]>

* Bump github actions versions #75

Update the following actions:

* actions/checkout
* actions/setup-python

Reference: aboutcode-org/skeleton#75
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Update github actions

Reference: aboutcode-org/skeleton#75
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Update RTD buil

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Fix unordered lists issue

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add redirects for docs

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Create script to update repo skeleton #80

Signed-off-by: Jono Yang <[email protected]>

* Add macOS-13 job in azure-pipelines.yml

Signed-off-by: Jono Yang <[email protected]>

* Added docs server script, dark mode & copybutton for docs

Signed-off-by: Omkar Phansopkar <[email protected]>

* Update CSS to widen page and handle mobile #84

Reference: aboutcode-org/skeleton#84

Signed-off-by: John M. Horan <[email protected]>

* Delete theme_overrides_SUPERSEDED.css as no longer needed #84

Reference: aboutcode-org/skeleton#84

Signed-off-by: John M. Horan <[email protected]>

* Fix top padding for rst content

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Update CI runners and python version

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Replace deprecated macos CI runners

Replace macos-11 runners with macos-14 runners.

Reference: https://github.com/actions/runner-images?tab=readme-ov-file#available-images
Reference: aboutcode-org/skeleton#89
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Update minimum required python version to 3.8

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Update link references of ownership from nexB to aboutcode-org

Signed-off-by: Chin Yeung Li <[email protected]>

* Drop deprecated macos-12 runner

* Update CI/Actions runners

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Replace black and isort with ruff

    * Use ruff config and Make commands from scancode.io

Signed-off-by: Jono Yang <[email protected]>

* Use org standard 100 line length

Signed-off-by: Philippe Ombredanne <[email protected]>

* Lint all common code directories

Signed-off-by: Philippe Ombredanne <[email protected]>

* Remove unused targets

Signed-off-by: Philippe Ombredanne <[email protected]>

* Improve import sorting

Signed-off-by: Philippe Ombredanne <[email protected]>

* Apply small code updates

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Apply cosmetic refactorings

Signed-off-by: Philippe Ombredanne <[email protected]>

* Reformat test code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine ruff configuration

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format doc

Signed-off-by: Philippe Ombredanne <[email protected]>

* Run doc8 on all rst files

Signed-off-by: Philippe Ombredanne <[email protected]>

* Enable doc style checks

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add support for new OS versions

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update scripts aboutcode references

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not format more test data

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not treat rst as Python

Signed-off-by: Philippe Ombredanne <[email protected]>

* Combine testing and docs extra for simplicity

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine checking of docs with doc8

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine doc handling

* remove CI scripts and use Makefile targets instead
* ensure doc8 runs quiet
* add new docs-check make target to run documentation and links checks
* update oudated doc for docs contribution

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add twine check to release publication

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine doc contribution docs

Signed-off-by: Philippe Ombredanne <[email protected]>

* Replace black and isort with ruff

    * Use ruff config and Make commands from scancode.io

Signed-off-by: Jono Yang <[email protected]>

* Use org standard 100 line length

Signed-off-by: Philippe Ombredanne <[email protected]>

* Lint all common code directories

Signed-off-by: Philippe Ombredanne <[email protected]>

* Remove unused targets

Signed-off-by: Philippe Ombredanne <[email protected]>

* Improve import sorting

Signed-off-by: Philippe Ombredanne <[email protected]>

* Apply small code updates

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Apply cosmetic refactorings

Signed-off-by: Philippe Ombredanne <[email protected]>

* Reformat test code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine ruff configuration

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format doc

Signed-off-by: Philippe Ombredanne <[email protected]>

* Run doc8 on all rst files

Signed-off-by: Philippe Ombredanne <[email protected]>

* Enable doc style checks

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not format more test data

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not treat rst as Python

Signed-off-by: Philippe Ombredanne <[email protected]>

* Combine testing and docs extra for simplicity

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine checking of docs with doc8

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine doc handling

* remove CI scripts and use Makefile targets instead
* ensure doc8 runs quiet
* add new docs-check make target to run documentation and links checks
* update oudated doc for docs contribution

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add twine check to release publication

Signed-off-by: Philippe Ombredanne <[email protected]>

* Refine doc contribution docs

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update codestyle command

    * Remove trailing whitespace

Signed-off-by: Jono Yang <[email protected]>

* Update README.rst

Signed-off-by: Jono Yang <[email protected]>

* Update BUILDDIR envvar in docs/Makefile

Signed-off-by: Jono Yang <[email protected]>

* Fix doc line length

Signed-off-by: Philippe Ombredanne <[email protected]>

* Format code

Signed-off-by: Philippe Ombredanne <[email protected]>

* Correct supported runner on Azure

See for details:
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml

macOS ARM images do not seem to be supported there

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add code checks to CI

Remove running "make check" as a test

Signed-off-by: Philippe Ombredanne <[email protected]>

* Revert support for Python 3.13

This is not yet supported everywhere

Signed-off-by: Philippe Ombredanne <[email protected]>

* Ignore local .env file

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add correct extras for documentation

Signed-off-by: Philippe Ombredanne <[email protected]>

* Improve MANIFEST

Signed-off-by: Philippe Ombredanne <[email protected]>

* Improve cleaning on POSIX

Signed-off-by: Philippe Ombredanne <[email protected]>

* Rename dev extra to "dev"

Instead of testing ... and update references accordingly

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add more excludes from tests

Signed-off-by: Philippe Ombredanne <[email protected]>

* Do not lint django migrations

Signed-off-by: Philippe Ombredanne <[email protected]>

* Add README.rst to list of "license files"

Signed-off-by: Philippe Ombredanne <[email protected]>

* Use Python 3.9 as lowest suupported version

Signed-off-by: Philippe Ombredanne <[email protected]>

* Drop pycodestyle

Not used anymore

Signed-off-by: Philippe Ombredanne <[email protected]>

* Bump pytest minimal version

Signed-off-by: Philippe Ombredanne <[email protected]>

* Update CI runners and scripts

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add missing wheel builds on release CI

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Drop python3.9 support and add python 3.14

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Support trusted-publishing for package releases

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Update RTD build python version

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add initial code

Signed-off-by: Jono Yang <[email protected]>

* Add initial purl map

Signed-off-by: Jono Yang <[email protected]>

* Remove functions to create PurlValidator from purl list

    * This was done for dev purposes

Signed-off-by: Jono Yang <[email protected]>

* Remove unnecessary code

Signed-off-by: Jono Yang <[email protected]>

* Update comments

Signed-off-by: Jono Yang <[email protected]>

* Update AUTHORS.rst

Signed-off-by: Jono Yang <[email protected]>

* Update requirements.txt

Signed-off-by: Jono Yang <[email protected]>

* Add tests

Signed-off-by: Jono Yang <[email protected]>

* Update formatting

Signed-off-by: Jono Yang <[email protected]>

---------

Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Chin Yeung Li <[email protected]>
Signed-off-by: Jono Yang <[email protected]>
Signed-off-by: Tushar Goel <[email protected]>
Signed-off-by: keshav-space <[email protected]>
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Signed-off-by: swastik <[email protected]>
Signed-off-by: Arnav Mandal <[email protected]>
Signed-off-by: Arijit De <[email protected]>
Signed-off-by: Omkar Phansopkar <[email protected]>
Signed-off-by: John M. Horan <[email protected]>
Co-authored-by: Philippe Ombredanne <[email protected]>
Co-authored-by: Chin Yeung Li <[email protected]>
Co-authored-by: Tushar Goel <[email protected]>
Co-authored-by: Philippe Ombredanne <[email protected]>
Co-authored-by: keshav-space <[email protected]>
Co-authored-by: Ayan Sinha Mahapatra <[email protected]>
Co-authored-by: swastik <[email protected]>
Co-authored-by: Arnav Mandal <[email protected]>
Co-authored-by: Arijit De <[email protected]>
Co-authored-by: Omkar Phansopkar <[email protected]>
Co-authored-by: John M. Horan <[email protected]>
Co-authored-by: Keshav Priyadarshi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@tardyp tardyp tardyp left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pombredanne @tardyp @steven-esser