* Settings: add an option to skip checksum verification on Updates.xml
This allows aqt to work even when the server download.qt.io is
unreachable.
Signed-off-by: Alberto Mardegan <mardy@users.sourceforge.net>
* Update settings.ini
* Correct option to skip checksum verification
* Rename ignore_hash option to INSECURE_NOT_FOR_PRODUCTION_ignore_hash
* Update doc about INSECURE_NOT_FOR_PRODUCTION_ignore_hash
* Add hash_algorithm option
* Unofficial mirrors might not provide any hash for Updates.xml
* docs: update configuration.rst
express default value of hash_algorithm
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* fix: hashlib constructor with usedforsecurity=False
- Python 3.9 and later introduce a keyword argument ``usedforsecurity``
- Set to False because we use hash to check file integrity not for password hash.
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Attempt to fix parameter type inconsistency in downloadBinaryFile
---------
Signed-off-by: Alberto Mardegan <mardy@users.sourceforge.net>
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
Co-authored-by: Alberto Mardegan <mardy@users.sourceforge.net>
Co-authored-by: Hiroshi Miura <miurahr@linux.com>
- Replace map+filter and list+filter with list comprehension expression
- Use try/except for next(Iterator)
- Add test case for altlink that cause exception on next() in get_altlink
- Avoid exit but use sys.exit
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
- Introduce classes inherit argparse.ArgumentParser to suppress mypy
errors in Cli class.
- `requests.session.Session.get()` expects that `timeout` is `float | tuple[float, float] | tuple[float, None] | None`
so use `Tuple[float, float]` for timeout
- Use named tuple for modules_query
- Update test_list.py
Causes `get_hash` to verify the hash length, and check that the hash can
be unhexlified properly.
Fixes the interface for `get_hash` so that the caller does not have to
run `binascii.unhexlify` or verify its output.
I don't think we can really call the `Updates.xml` files that we are
parsing 'untrusted xml', because we are checking that they match the
sha256 checksum. However, I don't think there's any good reason not to
use a more secure parsing algorithm.
aqt.helper.MyConfigParser.getlist fails to retrieve a fallback list when
the `section` parameter does not exist in the `settings.ini` file.
This ensures that the fallback is used when that key is missing.
To keep this commit small, `hashurl` was removed from QtPackage, and
`get_hash` constructs the hash url based on the url of the 7z archive
to download. I think that in the future, QtArchive and QtPackage could
be refactored to construct this url more appropriately. However, this
would be a complicated change that doesn't belong in this commit.
getUrl() now use `requests.get()` that is default getter that does not retry.
This change corrects it to `session.get()`
Related issue #472
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
This prevents `list-*` from printing any module whose uncompressed size
is 40 bytes or less.
I have found a couple of modules that are exactly 40 bytes, and they
include no files, only empty directories. If any user tries to install
these modules, they will think that aqt failed to install the module
without printing any error message. I prefer to avoid this issue
by preventing `aqt list-*` from printing them.
`aqt install-*` is unaffected by this change, and it can still install
modules that contain no files. It is likely that we will need to print
a warning when this occurs.
If any user disagrees with the threshold value of 40 bytes, they are
free to adjust that value to their liking in `settings.ini`.
This allows a user to turn on `--keep` every time they run the program,
and to set the default archive destination when `--keep` is turned on.
Using `--archive-dest` overrides the archive destination in setting.ini.
This adds the parameter `archive_dest` to `run_installer`, which will
control where `helper.downloadBinaryFile` will download files, and sets
it to "." by default.
For convenience/readability, this will also turn any reference to the
downloaded file into a `pathlib.Path` object. This makes it easier to
ensure that the file ends up in the right location.
This change deprecates the `max_retries` option in settings.ini,
replacing it with `max_retries_on_connection_error`, in order to
differentiate it from `max_retries_on_checksum_error`.
This change separates the handling of hash mismatches from all other
exceptions.
The functions in the `try` block (open, iter_content, write, update,
flush) can generate a wide variety of exceptions with no common base
class other than Exception. It would be my preference to differentiate
between file IO exceptions, hash update errors, and input stream errors,
but it seems impractical to track down all the different types. For that
reason, I am wrapping the base Exception type in an
ArchiveDownloadError. I am removing the `exc=sys.exc_info()` and
`logger.error(exc[1])` because it's redundant. `sys.exc_info()[1]`
returns the current exception being handled, which is `e`.
* Rename duplicate labels
When using 'sphinx.ext.autosectionlabel', labels are automatically
generated with the same name as the title of each section.
This change renames the manually created labels so that they do not
clash with the autogenerated labels, and silences the warnings
triggered by them.
* Extend title underlines
This silences the warning about the title underlines being too short.
* Add blank line after "code-block" directive
Silences this warning:
WARNING: Error in "code-block" directive: maximum 1 argument(s) allowed, 9 supplied.
* Add blank line where required by sphinx docstring
Silences this warning: WARNING: Unexpected indentation.
* Revert "Rename duplicate labels"
This reverts commit 442d2d81
* Add explicit labels for `cli.rst` sections
* Disable sphinx.ext.autosectionlabel
* Turn off html static path
The '_static' path is not being used, and it doesn't even exist, which
triggers a warning. This change silences that warning.
* Fix label for 'qt installation command'
This adds the function `ListCommand.fetch_tool_by_simple_spec`. This
allows the caller to specify a SimpleSpec and the name of a tool, and
the function will return the Updates.xml record for the specified tool
with the largest version number that satisfies the SimpleSpec. This is
meant to allow users to install tools without specifying an explicit
version number.
This also adds some test code to validate the complicated parts of this
code.
This also extends `helper.xml_to_modules`, so that a caller can
automatically get all tags in an XML element without explicitly
specifying them.
* Avoid calling functions with positional args: use names in case you reorder, Some type hints + __repr__ and __str__
* Use posixpath.join
* mypy issue: logger is sometimes passed.
* mypy: rename the Settings class to SettingsClass
```
aqt/helper.py:333: error: Cannot assign to a type
aqt/helper.py:333: error: Incompatible types in assignment (expression has type "Settings", variable has type "Type[Settings]")
```
* Reduce indenting by avoiding unnecessary except.else
* more calling by named arguments
* Add html scraping capability to helper.py
The function scrape_html_for_folders() reads an html file at
`https://download.qt.io/online/qtsdkrepository/<os>/<target>/` and
extracts a list of all the folders within that directory.
It separates the folders by category, so that 'tools' are not
mixed in with 'qt5' or 'qt6'.
This commit also adds a function that prints the available folders,
allowing the caller to filter the output by category.
* Add interface for accessing scrape_html_for_folders
I'm sure this isn't exactly what is needed, and that this functionality
should probably be folded into `aqt list`, but it does demonstrate that
aqt can list all the tools available at downloads.qt.io by scraping html.
This allows you to type `aqt list2 tools linux desktop`, and get a list
of all the tools on the server.
There are more options; type `aqt list2 -h` to see them all.
* Add failing tests for `aqt list`
* Add passing implementation of `aqt list`
* Remove old implementation of `aqt list`
This removes `Cli.run_list_old`, as well as any commented-out code in
installer.py that was related to the old implementation of `run_list`.
This also restores some code that was commented out erroneously.
* Fix misuse of semantic_version
In an earlier commit, I used the parameter `partial=True` for the
preview builds of Qt, since they do not include a patch number. I did
not recognize this as a misuse of semantic_version until I ran the tox
linter. `partial` is meant for 'partial specification', as in, 'this
program requires version 6.2, and the patch is unimportant`.
This change removes the `partial=True` parameter, adds a `0` for the
patch number, and uses a format string to print preview versions without
the patch.
* Use semantic version in interfaces instead of str
This updates all functions that accept a Qt version as a string, so that
they now accept a semantic version object instead. This forces the
caller to do some input validation, and moves the responsibility of
string-to-version conversions out of functions where it does not belong.
* Improve `aqt list` interface as discussed
This adds several new options to `aqt list`, and renames the ambiguous
`packages` to `modules`. New options include the ability to list
targets, extensions, and architectures for a particular (host, target,
Qt version) tuple. The goal is to provide sufficient information to the
user, such that the user can use all the other features of aqt without
having to look them up somewhere else.
* Cleanup `targets` dictionary
* Add `list architectures` feature
* Add tests for `list architectures` feature
* Add documentation for new list feature.
* Fix url bugs related to BASE_URL change
* Add stub for `list --extensions`
* Implement `list --extensions`
* Cleanup imports
* Add basic tests of `aqt list` to azure pipeline
* Fix failed merge with upstream
* Simplify string-to-version code
An earlier commit introduced `cli_2_semantic_version`, which calls
`get_semantic_version_with_dots`, which will never be called by any
other code. I think that this should really just be one function, and
splitting it in two is just needlessly complex.
* Revert dd41e653 - changes to azure-pipelines.yml
* CI: change test case for list command
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Update MANIFEST
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Fix errors reported by flake8
This commit fixes some unused imports, missing imports, and formatting
errors.
* Run isort
* Add three more build jobs to azure pipelines
This adds three more combinations of `qt version/host/target` to the
`aqt list` command tests on the azure pipelines. This is pretty far from
exhaustive, but I think it's worth it to add these: they execute
slightly different code paths than the standard `linux/desktop`
variants. I don't want to add a whole lot more than this; `aqt list` is
slower than I'd like it to be, and I don't want to waste CI time.
* Fix bug where empty extensions returns success
Prior to this commit, if you ran `aqt list` to list possible extensions,
it could print a blank line and return 0. This is because the
get_extensions_for_version function was accepting empty strings as a
valid extension, and returning a list consisting of one empty string.
This commit filters out those empty strings.
* Attempt to fix ci bug when listing android exts
Android extensions for Qt6 follow different rules than Qt5; this
attempts to account for that.
* remove invalid ArchiveId objects from tests
* Add author
* Use Python for test logic instead of bash
* Remove 'build accelbubble' step from 'list' test
This commit attempts to separate the 'build accelbubble example' test
from the 'aqt list' test.
In this case, the test for `aqt list qt5 linux android` was running the
"build accelbubble example" test in addition to the intended test. If
the "build accelbubble example" test fails, then it looks like the `aqt
list` test failed, until you look to see why it failed.
* Remove dead code
* Accept `latest` keyword instead of qt version
This commit modifies the interface of `aqt list`, so that the flags
`--modules`, `--extensions`, and `--arch` will accept either a Qt
version or the word `latest`. If the user specifies a version of Qt,
the command works as before. If the user specifies `--modules latest`,
`aqt list` will determine the latest version of Qt, given the filters
specified, and return the modules for that version.
This commit removes the flag `--latest-modules`, because that flag is no
longer necessary. The `latest` keyword serves the same function, and it
works for the `--extensions`, and `--arch` flags as well. This prevents
the need for flags such as `--latest-extensions` and `--latest-arch`,
which otherwise would have been necessary to provide this functionality
to the other list features.
* remove unused import
* Update azure pipeline to reflect `latest` feature
* WIP refactor helper methods into ListCommand
* Refactor helper methods into ListCommand
This PR adds many helper methods to helper.py that are closely related
to each other, and which are only useful to the ListCommand object. This
commit moves the majority of these into the ListCommand object, so that
they don't confuse future readers who are looking for something else.
* Fix imports after merge
* Fix unused import
* Add test and impl for list tool modules
This adds code that parses an 'Updates.xml' file for the names of tool
modules, which a user can use to install tools. This does not have any
cli-facing code yet, since the current interface does not make sense for
this purpose.
* Add CLI command to list tool variant names
This exposes `ListCommand.fetch_tool_modules` to the CLI.
Accordingly, it adds a test to the Azure Pipelines for every `aqt list`
job, and it adds documentation to docs/cli.rst.
Since this feature is meant to print values you can feed into the `aqt
tool` command, the documentation for the `aqt tool` command was changed
as well.
* Use monkeypatch for http_fetcher
This removes the http_fetcher parameter for ListCommand in favor of
monkeypatching.
* Remove dataclasses
Dataclasses were new in Python 3.7, and don't exist in the minimum
supported Python 3.6.
* black
* Fix ci test for `aqt list tools --tool`
* Add default timeout to request_http_with_failover
* fix monkeypatched http fetcher
* Fix 'aqt list tools' command on Azure Pipeline
Currently, the `tools_qt3dstudio_runtime_240` appears to be the only
tool available on all host/target combinations, so we will check that
one in Azure Pipelines.
* Move helper classes specific to ListCommand into ListCommand
* Fix use of new global Settings object
* Fix sphinx docstring syntax
* Swap `request_http_with_failover` for `getUrl`
* Add additional follow-up suggestion on error
* Fix exceptions import
* Prevent repeated 'suggested follow up' actions
This change uses exclusive branches to prevent `aqt list` from repeating
the same/similar recommended actions.
* Fix bugs related to updated getUrl error messages
* run black
* docs: add auto section link label extension
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* docs: add more list command examples
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Fix typo in docs
* remove duplicate `semantic_version`
* Add expected outputs and arch example to docs
This change adds some expected output to the documentation. For any
user who is confused about what the documentation means when it
mentions `tools` and `modules`, and is unsure which is which, this
change should help that user figure that out.
This also adds an example for listing architectures.
* Remove `logger` from `getUrl` parameter list
* helper: getLogger("aqt.helper") in helper functions
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Use `pretty_print` instead of `str()` for output
* test_list.py use pretty_print instead of str
* fix Versions constructor
* black
Co-authored-by: Hiroshi Miura <miurahr@linux.com>
The original error message, `Download error when access <filename>`,
is confusing when a user is trying to use `aqt list`. This message
could make the user think that `aqt list` is attempting to download
archives, when in reality the command is only reading files on the
server.
Initially, I wanted to make this error message modifiable by the caller.
On further investigation, I realized that this function is always used
to retrieve 'Updates.xml' files or hashes, and that the error message
"Failed to retrieve file" would be appropriate for all usages.
Admittedly, the distinction between "Download error" and "Failed to
retrieve file" is subtle. However, the difference was big enough to
convince me as a developer that I should not use this function to
retrieve "Updates.xml" files, when in fact it was the right function to
use.
This change handles multiple redirects. Without this change, if
`helper.altlink()` returns a url that will redirect, we cannot follow
that redirect. This change allows up to 10 such redirects to occur.
Please note that the upper bound for number of redirects was chosen
arbitrarily, with no regard for what might realistically occur. I felt
that it was important not to allow recursive or infinite redirects, and
that at least two redirects should be allowed. Perhaps the upper bound
should be modifiable in settings.ini.
It apply KDE patch collection to Qt 5.15.2 source
`aqt src 5.15.2 linux desktop --kde`
Patch collection is based on Qt 5.15.2, so only the version can be patched.
Ref #231
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Use global variable to hold Settings
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Settings global instance to hold configurations
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Revert to Settings class to helper module
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Revert Settings class variables
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* PEP8
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* CI: change actions to use macos-11
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
* Revert "CI: change actions to use macos-11"
CI: macos-11 is just for private preview
This reverts commit f1d576f4bf.
