Commit Graph

196 Commits

Author SHA1 Message Date
David Dalcino
8aa7af2524 Add new syntax to installer.Cli 2021-07-22 09:53:22 -07:00
Hiroshi Miura
0c346d77c8 Merge branch 'master' into topic-tool-latest 2021-07-15 14:10:09 +09:00
Hiroshi Miura
15267182fa Refactoring class structure: Factory class, Data and consumer (#309)
* Introduce tooldata class

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Refactoring list command

- Introduce show_list function that is part of UI
- ListCommand class can be used as library to retrieve metadata.
- Rename ListCommand to MetadataFactory
- Rename action method to getList

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-15 11:48:39 +09:00
Hiroshi Miura
62d8ed3b55 Rename to get_packages
- It is problematic name that QtArchive.get_archives returns List[QtPackage]

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-15 09:32:26 +09:00
Hiroshi Miura
f1b8b5a2c3 Refactoring call_instaler/installer
- Introduce `run_installer` that consume `List[QtArchives]` data
  to be same structure as `show_list` and `ToolData` data class.

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-15 08:33:50 +09:00
Hiroshi Miura
ab823596dc Utilize dataclasses (#313)
* Utilize dataclasses

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-15 08:08:53 +09:00
Hiroshi Miura
a5bb6b7e96 Introduce metadata package (#305)
* Introduce metadata module

that has Versions and ListCommand class

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Introduce custom Version class

A Version class override semantic_version.Version class
that accept qt versions and tools versions

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8/Black

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Improve test assertion

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Move ArchiveId class to metadata module

And Improve imports

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Refactoring Version class

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Refactoring Table class

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Refactoring Table class(wip)

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix output format of `list tools` command

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* list: Improve print output

avoid str.join()

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-13 00:10:36 +09:00
Hiroshi Miura
fff94542cf PEP8/Black
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-11 11:17:53 +09:00
Hiroshi Miura
1846e2edcf Install all listed tools
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-11 11:07:11 +09:00
Hiroshi Miura
3d9ef1ac6b Update dummy version handling
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-11 10:29:29 +09:00
David Dalcino
10d4ec02d5 Add tool-long-listing to ListCommand
This change adds the ability to list all the 'tool variant names'
alongside the version and release date of the tool.

When using the `aqt list tools` feature, I often find that I need more
information than the tool provides. I often need to know the release
date and the version for each tool, and I can't get that information
without looking up the Updates.xml file. This feature enables me to skip
that part.
2021-07-09 19:05:08 -07:00
Hiroshi Miura
358687d7e5 Merge branch 'master' into topic-tool-latest 2021-07-10 09:29:22 +09:00
Hiroshi Miura
62c7075aeb Accept target argument 2021-07-09 23:53:41 +09:00
David Dalcino
860d297b2d Validate version strings more consistently 2021-07-06 12:16:58 -07:00
Hiroshi Miura
f007ad8e7b Quick hack to install latest version of tool
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-06 18:15:21 +09:00
Hiroshi Miura
47f312e9e8 Tool subcommand show aqt version in log
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-06 13:22:01 +09:00
Julien Marrec
91f33da898 Minor changes: use posixpath.join, add __repr__/__str__, some type hints, and some mypy issues (#287)
* Avoid calling functions with positional args: use names in case you reorder, Some type hints + __repr__ and __str__

* Use posixpath.join

* mypy issue: logger is sometimes passed.

* mypy: rename the Settings class to SettingsClass

```
aqt/helper.py:333: error: Cannot assign to a type
aqt/helper.py:333: error: Incompatible types in assignment (expression has type "Settings", variable has type "Type[Settings]")
```

* Reduce indenting by avoiding unnecessary except.else

* more calling by named arguments
2021-07-06 12:13:35 +09:00
David Dalcino
705c6225e2 Add capability to list tools (#235)
* Add html scraping capability to helper.py

The function scrape_html_for_folders() reads an html file at
`https://download.qt.io/online/qtsdkrepository/<os>/<target>/` and
extracts a list of all the folders within that directory.
It separates the folders by category, so that 'tools' are not
mixed in with 'qt5' or 'qt6'.

This commit also adds a function that prints the available folders,
allowing the caller to filter the output by category.

* Add interface for accessing scrape_html_for_folders

I'm sure this isn't exactly what is needed, and that this functionality
should probably be folded into `aqt list`, but it does demonstrate that
aqt can list all the tools available at downloads.qt.io by scraping html.

This allows you to type `aqt list2 tools linux desktop`, and get a list
of all the tools on the server.
There are more options; type `aqt list2 -h` to see them all.

* Add failing tests for `aqt list`

* Add passing implementation of `aqt list`

* Remove old implementation of `aqt list`

This removes `Cli.run_list_old`, as well as any commented-out code in
installer.py that was related to the old implementation of `run_list`.

This also restores some code that was commented out erroneously.

* Fix misuse of semantic_version

In an earlier commit, I used the parameter `partial=True` for the
preview builds of Qt, since they do not include a patch number. I did
not recognize this as a misuse of semantic_version until I ran the tox
linter. `partial` is meant for 'partial specification', as in, 'this
program requires version 6.2, and the patch is unimportant`.

This change removes the `partial=True` parameter, adds a `0` for the
patch number, and uses a format string to print preview versions without
the patch.

* Use semantic version in interfaces instead of str

This updates all functions that accept a Qt version as a string, so that
they now accept a semantic version object instead. This forces the
caller to do some input validation, and moves the responsibility of
string-to-version conversions out of functions where it does not belong.

* Improve `aqt list` interface as discussed

This adds several new options to `aqt list`, and renames the ambiguous
`packages` to `modules`. New options include the ability to list
targets, extensions, and architectures for a particular (host, target,
Qt version) tuple. The goal is to provide sufficient information to the
user, such that the user can use all the other features of aqt without
having to look them up somewhere else.

* Cleanup `targets` dictionary

* Add `list architectures` feature

* Add tests for `list architectures` feature

* Add documentation for new list feature.

* Fix url bugs related to BASE_URL change

* Add stub for `list --extensions`

* Implement `list --extensions`

* Cleanup imports

* Add basic tests of `aqt list` to azure pipeline

* Fix failed merge with upstream

* Simplify string-to-version code

An earlier commit introduced `cli_2_semantic_version`, which calls
`get_semantic_version_with_dots`, which will never be called by any
other code. I think that this should really just be one function, and
splitting it in two is just needlessly complex.

* Revert dd41e653 - changes to azure-pipelines.yml

* CI: change test case for list command

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Update MANIFEST

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix errors reported by flake8

This commit fixes some unused imports, missing imports, and formatting
errors.

* Run isort

* Add three more build jobs to azure pipelines

This adds three more combinations of `qt version/host/target` to the
`aqt list` command tests on the azure pipelines. This is pretty far from
exhaustive, but I think it's worth it to add these: they execute
slightly different code paths than the standard `linux/desktop`
variants. I don't want to add a whole lot more than this; `aqt list` is
slower than I'd like it to be, and I don't want to waste CI time.

* Fix bug where empty extensions returns success

Prior to this commit, if you ran `aqt list` to list possible extensions,
it could print a blank line and return 0. This is because the
get_extensions_for_version function was accepting empty strings as a
valid extension, and returning a list consisting of one empty string.
This commit filters out those empty strings.

* Attempt to fix ci bug when listing android exts

Android extensions for Qt6 follow different rules than Qt5; this
attempts to account for that.

* remove invalid ArchiveId objects from tests

* Add author

* Use Python for test logic instead of bash

* Remove 'build accelbubble' step from 'list' test

This commit attempts to separate the 'build accelbubble example' test
from the 'aqt list' test.

In this case, the test for `aqt list qt5 linux android` was running the
"build accelbubble example" test in addition to the intended test. If
the "build accelbubble example" test fails, then it looks like the `aqt
list` test failed, until you look to see why it failed.

* Remove dead code

* Accept `latest` keyword instead of qt version

This commit modifies the interface of `aqt list`, so that the flags
`--modules`, `--extensions`, and `--arch` will accept either a Qt
version or the word `latest`. If the user specifies a version of Qt,
the command works as before. If the user specifies `--modules latest`,
`aqt list` will determine the latest version of Qt, given the filters
specified, and return the modules for that version.

This commit removes the flag `--latest-modules`, because that flag is no
longer necessary. The `latest` keyword serves the same function, and it
works for the `--extensions`, and `--arch` flags as well. This prevents
the need for flags such as `--latest-extensions` and `--latest-arch`,
which otherwise would have been necessary to provide this functionality
to the other list features.

* remove unused import

* Update azure pipeline to reflect `latest` feature

* WIP refactor helper methods into ListCommand

* Refactor helper methods into ListCommand

This PR adds many helper methods to helper.py that are closely related
to each other, and which are only useful to the ListCommand object. This
commit moves the majority of these into the ListCommand object, so that
they don't confuse future readers who are looking for something else.

* Fix imports after merge

* Fix unused import

* Add test and impl for list tool modules

This adds code that parses an 'Updates.xml' file for the names of tool
modules, which a user can use to install tools. This does not have any
cli-facing code yet, since the current interface does not make sense for
this purpose.

* Add CLI command to list tool variant names

This exposes `ListCommand.fetch_tool_modules` to the CLI.
Accordingly, it adds a test to the Azure Pipelines for every `aqt list`
job, and it adds documentation to docs/cli.rst.

Since this feature is meant to print values you can feed into the `aqt
tool` command, the documentation for the `aqt tool` command was changed
as well.

* Use monkeypatch for http_fetcher

This removes the http_fetcher parameter for ListCommand in favor of
monkeypatching.

* Remove dataclasses

Dataclasses were new in Python 3.7, and don't exist in the minimum
supported Python 3.6.

* black

* Fix ci test for `aqt list tools --tool`

* Add default timeout to request_http_with_failover

* fix monkeypatched http fetcher

* Fix 'aqt list tools' command on Azure Pipeline

Currently, the `tools_qt3dstudio_runtime_240` appears to be the only
tool available on all host/target combinations, so we will check that
one in Azure Pipelines.

* Move helper classes specific to ListCommand into ListCommand

* Fix use of new global Settings object

* Fix sphinx docstring syntax

* Swap `request_http_with_failover` for `getUrl`

* Add additional follow-up suggestion on error

* Fix exceptions import

* Prevent repeated 'suggested follow up' actions

This change uses exclusive branches to prevent `aqt list` from repeating
the same/similar recommended actions.

* Fix bugs related to updated getUrl error messages

* run black

* docs: add auto section link label extension

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* docs: add more list command examples

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix typo in docs

* remove duplicate `semantic_version`

* Add expected outputs and arch example to docs

This change adds some expected output to the documentation. For any
user who is confused about what the documentation means when it
mentions `tools` and `modules`, and is unsure which is which, this
change should help that user figure that out.

This also adds an example for listing architectures.

* Remove `logger` from `getUrl` parameter list

* helper: getLogger("aqt.helper") in helper functions

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Use `pretty_print` instead of `str()` for output

* test_list.py use pretty_print instead of str

* fix Versions constructor

* black

Co-authored-by: Hiroshi Miura <miurahr@linux.com>
2021-07-06 07:58:18 +09:00
Hiroshi Miura
265c0c5b4b Change log level to debug for settings.ini loading
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-07-05 13:04:04 +09:00
David Dalcino
5d5bae28ab run black 2021-07-01 12:04:37 -07:00
David Dalcino
545484a93d Implement version validation 2021-07-01 12:02:47 -07:00
Hiroshi Miura
f35edf398a Fix logging in installer sub-process when multiprocessing context with spawn method (#273)
* Simplify logging

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Update logging

- keep logging file path in Settings
- introduce installer logger

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8/Black

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Logging: add queue handler

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Logging: introduce logutils module

- Add QueueListenerHandler
- Add queue listener handler config
- qualname to aqt.installer

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Logging: introduce aqt.updater logger

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Logging: introduce aqt.main and aqt.archives logger

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8/Black

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Logging: simplify config file

- use propergate for last output handler
- change formatter
  * console: brief
  * file: verbose
- Use multiprocessing.Queue() instead of queue.Queue()
- Remember logging config file customization in Settings
- QueueListener use handler as of getLogger("aqt").handlers

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* change log level of redirection

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Logging: Single global LoggingQueueListener

- Global instance of LoggingQueueListener
- Drop queue_listener from logging.ini
- Generate QueueHandler from LoggingQueueLisnter.get_queue_handler
- Explicitly close QueueListenerHandler and Queue object.

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* CI: use verbose console logging for ci

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* CLI: drop options to specify logger and logger configuration file

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Make Instance of LoggingQueueListener in Cli class

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Handle QueueListener in Cli.call_installer

- Use mp.Manager().Queue()
- start(), stop() in caller
- Gracefully close QueueHandler in installer

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Drop berkeley.edu from fallbacks of mirror site

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix log format of time digits

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Update settings load

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* CI: catch custom settings from change in master branch

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Show log when loading custom settings

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Improve subprocess logging that use root logger

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* fix logging.ini settings

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix against tests

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Introduce MyQueueListener

- Inherit System QueueListner to override logger name

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8/Black

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-06-21 15:25:00 +09:00
Hiroshi Miura
0776b54518 PEP8/Black
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-06-19 19:17:42 +09:00
Hiroshi Miura
99b9c216a5 Inroduce --kde option for src subcommand
It apply KDE patch collection to Qt 5.15.2 source
`aqt src 5.15.2 linux desktop --kde`

Patch collection is based on Qt 5.15.2, so only the version can be patched.

Ref #231

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-06-19 15:02:08 +09:00
Hiroshi Miura
fac014216f Use global instance to hold Settings instead of Borg (#272)
* Use global variable to hold Settings

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Settings global instance to hold configurations

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Revert to Settings class to helper module

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Revert Settings class variables

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* CI: change actions to use macos-11

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Revert "CI: change actions to use macos-11"

CI: macos-11 is just for private preview

This reverts commit f1d576f4bf.
2021-06-19 10:43:19 +09:00
Hiroshi Miura
fe38fb0e62 Version comparison by semantic_version (#263)
* Version comparison by packaging.version

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Use semantic_version instead of packaging.version

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Use more semantic_version

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix typo and imports

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Fix missing imports

and fix typo, reduce redundancies.

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* PEP8

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Add 5.12.11 to known version

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-06-12 10:46:45 +09:00
Hiroshi Miura
49fc8ec6b1 Version subcommand
- Set aqt version by version.py generation
- Drop importlib and pkg_resources chunks of code

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-06-11 08:34:43 +09:00
Hiroshi Miura
b066a8654c Drop feature to install old versions
The feature is infeasible to maintain, so drop the feature from release.

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-24 11:27:34 +09:00
Hiroshi Miura
741eb4989e PEP8/Black
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-24 08:27:07 +09:00
Hiroshi Miura
fb088cac05 settings.ini: fix -c option handling
when argparse.FileType("r") used, variable passed is a open-ed file.

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-23 15:16:38 +09:00
Hiroshi Miura
827d8d53e1 Introduce -c / --config command line option
option to specify settings.ini

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-23 14:35:05 +09:00
Hiroshi Miura
4c0638250c Move constants into settings.ini configuration
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-23 13:00:31 +09:00
Hiroshi Miura
ca72e070c7 test: Early load custom configuration
hack for borg class

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-23 12:12:04 +09:00
Hiroshi Miura
dd74254ea0 Check config file existence and handle parsing error
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-23 11:30:30 +09:00
Hiroshi Miura
6b12ce9e9d Merge pull request #244 from miurahr/patch-add-blacklist
Refactoring network code
2021-05-22 21:25:52 +09:00
Hiroshi Miura
43cb99bc9c Refactoring network code
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-22 17:26:16 +09:00
Hiroshi Miura
53c71ff664 refactor Cli class (#243)
* refactor Cli class

- check version combination
- remove dead code

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* fix typo in json

Signed-off-by: Hiroshi Miura <miurahr@linux.com>

* Update version combinations

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-22 14:25:17 +09:00
Hiroshi Miura
57cec9e2e0 Integrate cuteci installer into aqtinstall (#239)
* Add cuteci intaller

* Check qtaccount.ini existence

* Fix subprocess call

* sub command show elapsed time

* test: cli: update help message exception

* Update URL base handling

* old version installer accept environment variable

- qtaccount: QTLOGIN and QTPASSWORD variable

* Change cuteci sub command to 'offline_installer'

* Update qscript to deselect QtCreator from installation

* Reuse downloaded installer file

* Update test and fix MANIFEST

Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-20 14:43:02 +09:00
Hiroshi Miura
2bb8d68d6c PEP8
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-11 16:20:08 +09:00
Hiroshi Miura
c313b13c2a Check sha1 checksum of archive if possible
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-11 16:03:49 +09:00
David Dalcino
451bbb46ef Add RawTextHelpFormatter to install_parser
The `aqt install -h` command prints the help text for the `arch`
argument all together on one line. I noticed that the help text
is formatted very carefully in code, with lots of meaningful
whitespace, but on my computer, all that whitespace is being ignored.

With `formatter_class=argparse.RawTextHelpFormatter` added to
the `subparsers.add_parser()` function call, all of the whitespace
in the help text is preserved.

It looks like `formatter_class=argparse.RawTextHelpFormatter`
already exists in the call that creates the root parser, but it
appears that this property does not extend to all subparsers
automatically.
2021-05-10 15:33:33 -07:00
Hiroshi Miura
10dcbf6c5e Fix doc argument parse(#234)
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-05-02 11:48:15 +09:00
Hiroshi Miura
e5a062a150 Allow non-existence of py7zr dependency
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-04-08 11:35:37 +09:00
Hiroshi Miura
a697aeb88a PEP8
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-04-04 01:29:24 +09:00
Hiroshi Miura
33a5968d82 Merge branch 'master' into topic-keep-archive 2021-04-02 16:55:06 +09:00
Hiroshi Miura
d841741641 Reformat by black
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-04-02 12:20:10 +09:00
Hiroshi Miura
249108384a Drop importing inspect
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-03-31 22:48:09 +09:00
Hiroshi Miura
b20e9dcc44 Add --keep option to keep downloaded archives
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-03-15 14:41:55 +09:00
Hiroshi Miura
575613cec4 fix cli version error
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-02-25 20:58:18 +09:00
Hiroshi Miura
7ad2038711 Fix getting module version
Signed-off-by: Hiroshi Miura <miurahr@linux.com>
2021-02-25 20:49:53 +09:00