From e8eeea74e26d25cb805e72648a21ffc3ffd00f0f Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 22 Nov 2019 21:02:20 +0100 Subject: [PATCH 001/196] add reference to the query repository --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 39bf1b9..cd98f20 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,12 @@ This repo holds binaries for the CodeQL CLI. # Getting started 1. Go to the [Releases page](https://github.com/github/codeql-cli-binaries/releases). -1. Find the latest release, select Assets, and download the zip file containing the CLI. -1. Read the [CodeQL CLI documentation](https://help.semmle.com/codeql/codeql-cli.html). +2. Find the latest release, select Assets, and download the zip file containing the CLI. +3. You'll also want to clone https://github.com/Semmle/ql to get the CodeQL queries and support + libraries. Please take note of the + [set-up instructions](https://help.semmle.com/codeql/codeql-cli/procedures/get-started.html) + for placing it in a location where the CLI can find it. +4. Read the rest of the [CodeQL CLI documentation](https://help.semmle.com/codeql/codeql-cli.html). # License From f3a8ec1679e8db005d7773bdfb936fbd2205b7cc Mon Sep 17 00:00:00 2001 From: Aditya Sharad <6874315+adityasharad@users.noreply.github.com> Date: Fri, 22 Nov 2019 12:29:34 -0800 Subject: [PATCH 002/196] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cd98f20..dfadc91 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ This repo holds binaries for the CodeQL CLI. 1. Go to the [Releases page](https://github.com/github/codeql-cli-binaries/releases). 2. Find the latest release, select Assets, and download the zip file containing the CLI. -3. You'll also want to clone https://github.com/Semmle/ql to get the CodeQL queries and support +3. You'll also want to clone https://github.com/Semmle/ql to get the CodeQL queries and libraries. Please take note of the [set-up instructions](https://help.semmle.com/codeql/codeql-cli/procedures/get-started.html) for placing it in a location where the CLI can find it. From a2b3dd7fdd676f7c83753e35a634d322ab8ff784 Mon Sep 17 00:00:00 2001 From: Sam Lanning Date: Thu, 16 Jan 2020 11:13:00 -0800 Subject: [PATCH 003/196] Update license --- LICENSE.md | 8 ++++---- README.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/LICENSE.md b/LICENSE.md index 2cfc570..0824845 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -29,10 +29,10 @@ for any of the following: non-OSI-approved open source license, unless it is to perform academic research or to demonstrate the Software); or** - * **For automated analysis, continuous integration or continuous - delivery, whether as part of normal software engineering processes - or otherwise.** - + * **To generate CodeQL databases for or during automated analysis, + continuous integration or continuous delivery, whether as part + of normal software engineering processes or otherwise.** + Further, except (and only to the extent) permitted by applicable law or applicable third-party license, you will not (and have no right to): diff --git a/README.md b/README.md index dfadc91..1437008 100644 --- a/README.md +++ b/README.md @@ -18,4 +18,4 @@ This repo holds binaries for the CodeQL CLI. By downloading, you agree to the [GitHub CodeQL Terms & Conditions](https://securitylab.github.com/tools/codeql/license). -GitHub CodeQL can only be used on codebases that are released under an OSI-approved open source license, or to perform academic research. It can't be used for automated analysis, continuous integration or continuous delivery, whether as part of normal software engineering processes or otherwise. For these uses, [contact the sales team](https://enterprise.github.com/contact). +GitHub CodeQL can only be used on codebases that are released under an OSI-approved open source license, or to perform academic research. It can't be used to generate CodeQL databases for or during automated analysis, continuous integration or continuous delivery, whether as part of normal software engineering processes or otherwise. For these uses, [contact the sales team](https://enterprise.github.com/contact). From 0f40bcabdbcc1655d5a60b4805aaf50b0cd82846 Mon Sep 17 00:00:00 2001 From: Marco Gario Date: Sun, 3 May 2020 16:46:14 +0200 Subject: [PATCH 004/196] Change Semmle/ql to github/codeql --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1437008..61acc2a 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ This repo holds binaries for the CodeQL CLI. 1. Go to the [Releases page](https://github.com/github/codeql-cli-binaries/releases). 2. Find the latest release, select Assets, and download the zip file containing the CLI. -3. You'll also want to clone https://github.com/Semmle/ql to get the CodeQL queries and +3. You'll also want to clone https://github.com/github/codeql to get the CodeQL queries and libraries. Please take note of the [set-up instructions](https://help.semmle.com/codeql/codeql-cli/procedures/get-started.html) for placing it in a location where the CLI can find it. From 0b26cb598f98524db734baa36a1624acfb91bd6a Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Wed, 6 May 2020 12:07:46 +0200 Subject: [PATCH 005/196] new license for v2.1.2 --- LICENSE.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/LICENSE.md b/LICENSE.md index 0824845..3616ada 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -1,6 +1,5 @@ GitHub CodeQL Terms and Conditions ================================== - These GitHub CodeQL Terms and Conditions ("**Terms**") are a legal agreement between you (either as an individual or on behalf of an entity) and GitHub, Inc. regarding your use of the GitHub CodeQL @@ -16,22 +15,26 @@ The Software is licensed on a per user basis. You may use the Software for any of the following: * To perform analysis on any codebase that is released under an Open - Source Initiative (OSI)-approved open source software license; + Source Initiative (OSI)-approved open source software license (an + “Open Source Codebase”); + + * To generate CodeQL databases for or during automated analysis, + continuous integration (CI) or continuous delivery (CD) in the + following cases: (1) on any Open Source Codebase hosted and + maintained on GitHub.com, and (2) to test CodeQL queries you have + released under an OSI-approved open source software license to + confirm that new versions of those queries continue to find the + right vulnerabilities; * To perform academic research on any codebase; and * To demonstrate the Software. -**These Terms do not authorize, and the Software cannot be used:** - - * **For any purpose not expressly set forth above (for example, you - cannot use the Software on a code base that is released under a - non-OSI-approved open source license, unless it is to perform - academic research or to demonstrate the Software); or** - - * **To generate CodeQL databases for or during automated analysis, - continuous integration or continuous delivery, whether as part - of normal software engineering processes or otherwise.** +**These Terms do not authorize, and the Software cannot be used, for +any purpose not expressly set forth above, including to generate any +CodeQL database for or during automated analysis, CI or CD, whether as +part of normal engineering processes or otherwise, except in the +express cases set forth above.** Further, except (and only to the extent) permitted by applicable law or applicable third-party license, you will not (and have no right From cb1f162ed20d9ed54c92dffcf36aae2b605d871f Mon Sep 17 00:00:00 2001 From: James Fletcher <42464962+jf205@users.noreply.github.com> Date: Mon, 22 Jun 2020 16:56:47 +0100 Subject: [PATCH 006/196] Create CHANGELOG.md --- CHANGELOG.md | 361 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f68af8b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,361 @@ +# CodeQL CLI changelog + +## Release 2.2.3 (2020-06-15) + +CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new +version number because the `v2.2.2` folder on the download site +originally contained the 2.2.0 binaries instead of the correct 2.2.2 +ones. + +If you have downloaded release 2.2.2, and `codeql --version` correctly +identifies itself as being that version, you don't need to upgrade to +2.2.3. + +## Release 2.2.2 (2020-06-12) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.24) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.24 instance, you need to create them with release + 2.1.4. + +### Improvements + +- Query evaluations that time out due to a `--timeout` option are no + longer silently discarded. Instead `codeql` will terminate with exit + code 33. Commands that evaluate multiple queries will produce as + much output as they can even if one of the queries times out. + +## Release 2.2.1 + +There is no CodeQL CLI version 2.2.1. This version number was used +internally to work around restrictions in the CodeQL for VS Code +extension. + +## Release 2.2.0 (2020-05-29) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.24) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.24 instance, you need to create them with release + 2.1.4. +- Starting with this release, the CodeQL CLI can be downloaded either + as a single `codeql.zip` file containing the CLI for all supported + platforms, or as a `codeql-PLATFORM.zip` that contains the files for + just one platform. The single-platform zips are faster to download. + +### QL language improvement + +- QL now supports the definition of new types as type unions. This + feature currently allows unions of branches from an already existing + algebraic data type and unions of database types. + +## Release 2.1.4 (2020-05-26) + +This release corresponds to release 1.24.x of LGTM Enterprise, and +should be used when creating databases that will be uploaded to it. +Future CLI releases (numbered 2.2.x) may produce databases that are not +backwards compatible with this version of LGTM Enterprise. + +For all purposes other than creating databases for LGTM Enterprise we +recommend that you continue upgrading to newer CLI releases as they +become available. + +### Features added + +- A new `codeql query format` command exposes the QL autoformatter for + use on the command line. + +### Bugs fixed + +- `-J` command-line options that contain spaces now ought to work on + Windows. They still do not work reliably on Linux or MacOS, though. + +## Release 2.1.3 (2020-05-13) + +### Bugs fixed + +- Fixes a bug in `codeql execute cli-server` (a helper used by the VS + Code extension) which would sometimes cause query compilation to + fail until the extension was restarted. +- Fixes a bug in `codeql database upgrade` which could lead to + performance losses if the upgraded database was subsequently used + with LGTM or the legacy Semmle Core product. +- Fixes a bug in the QL evaluator that would sometimes lead to crashes + for queries that use the new `unique` aggregate added in release + 2.1.0. +- The value of the `--compilation-cache-size` option is now correctly + interpreted as a number of megabytes rather than a number of bytes. + +## Release 2.1.2 (2020-05-06) + +- Updated license terms to allow CI use with GitHub Actions for + open-source software. + +### Potentially breaking changes + +- In [query suite definitions](https://help.semmle.com/codeql/codeql-cli/procedures/query-suites.html), filter + instructions that filter on the `query path` pseudo-tag will now + always see the relative path to the query expressed with `/` as a + directory separator, independently of the platform. Previously they + erroneously used the platform's directory separator, meaning that + query suites developed on Windows would not work correctly on Unix + systems (and vice versa) if they used `query path`. Existing suite + definitions developed on Windows may need to be updated to match the + new behavior. + +### Features added + +- A new `codeql test accept` subcommand helps automate updating the + expected output for unit tests after a desired change in query + behavior. This can also be done by the new `--learn` option for + `codeql test run`. + +### Bugs fixed + +- `codeql database create` will now report an explicit error if given + a `--command` argument that specifies an empty string. Previously + this would be accepted initially, leading to confusing failures + later. + +## Release 2.1.1 (2020-04-20) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. + +### Features added + +- `codeql resolve queries` accepts a `--format=bylanguage` option. + This is used to help automated workflows determine which languages + to create databases for, from the queries that are available to run. +- It is now possible to attempt to execute `.ql` files that are not in + a QL pack. This is used by a few specialized internal workflows. + However, standalone queries cannot import any of the dependencies + that you would usually declare in a `qlpack.yml` file, so will not + be useful in most cases. + +## Release 2.1.0 (2020-03-27) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.23) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.23 instance, you need to create them with release + 2.0.1. For more information, see [Preparing CodeQL databases to + upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. + +### Potentially breaking changes + +- If you pass a directory name as a command-line argument to + `codeql test run`, it will now consider all `.ql` or `.qlref` files + found under that directory to be test queries, even if they have no + accompanying `.expected` file. Tests that lack an `.expected` file + will fail, but will generate an `.actual` file that you can rename + to `.expected` if you want to use the results. + + The goal of this change is to support existing workflows of + experienced CodeQL users, and also to provide clear error + indications if an `.expected` file is accidentally lost, renamed, or + misspelled. + + However, if you invoke `codeql test run` on a directory tree that + contains both tests and non-test queries, you will now encounter + errors if any of the `.ql` files can't be processed as test queries. + If you're affected by this change, you can suppress these errors by: + + - Adding a `tests` property to this QL pack to define specify + which directories contain only test queries and associated test + code. For more information, see + [About QL packs](https://help.semmle.com/codeql/codeql-cli/reference/qlpack-overview.html). + - Running `codeql test run` with a new `--strict-test-discovery` + option. + + In the longer term, we recommend that you reorganize the queries so + that test queries are stored in a directory tree that's separate + from actual queries. + +- `codeql database create` and `codeql database finalize` will no + longer recognize a `--no-duplicate-code` option. This option has + never had any effect, and its positive variant `--duplicate-code` + previously led to a fatal error. + +### Features added + +- A new XML extractor is included. It is not intended to be used as a + stand-alone extractor, but rather to augment the data produced by + other extractors. In particular, the C\# and Java extractors invoke + it during database creation to include information relevant to the + analysis of those languages, much like LGTM.com does. +- Two new plumbing commands `codeql database index-files` and + `codeql resolve files` have been added for support of invoking the + XML extractor support. These commands are generally only of interest + for extractor authors. +- Two new plumbing commands have been added to `codeql dataset`. The + `measure` subcommand can be used to collect size information from a + dataset, and the `check` subcommand can scan a dataset for database + inconsistencies. These commands are useful when developing a new + CodeQL extractor. +- The QL evaluator contains a number of features in support of an + internal experiment with using machine-learning techniques to + identify functions in unknown codebases as sources or sinks of + taint. This includes new command-line options `--ml-model-path` and + `--native-library-path` to several subcommands. As the new features + are not yet ready for general use, these new options should be + ignored by external CodeQL users. + +### Bugs fixed + +- Fixes a bug that could result in empty databases for C/C++. + Previously, extraction would mistakenly be skipped for source files + compiled with the Clang compiler, if the `-fintegrated-cc1` option + was specified. +- `codeql database create` and `codeql database init` will now, as + they have always been documented, refuse to create a database whose + parent directory doesn't already exist. +- `codeql test run` will no longer leave `.actual` files from previous + runs in the file system after a test passes. + +### QL language improvements + +- QL now supports set literals, and the QL extractor can identify them + with the `SetLiteral` class. For more information, see [Set literal + expressions](https://help.semmle.com/QL/ql-handbook/expressions.html#set-literal-expressions) + in the QL language reference. +- QL now supports a uniqueness aggregate. This can express constraints + that there is precisely one value. The syntax is taken from previous + aggregates such as `min` and `max`. + + ``` {.sourceCode .ql} + unique(int x | x = 4 or x = 2 * 2 | x) + ``` + +## Release 2.0.6 (2020-03-16) + +### Bugs fixed + +- Fixes a problem preventing `codeql database create` from working + with Python 3 on macOS. +- Fixes a problem preventing `codeql database create` from finding + locally installed Python packages. + +## Release 2.0.5 (2020-03-13) + +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the versions currently used on LGTM.com. These are newer than the + last release of LGTM Enterprise, so this release should not be used + if you plan to upload databases to an LGTM Enterprise instance. For + more information, see [Preparing CodeQL databases to upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. + +### Features added + +- `codeql test run` has a new `--slice` option that can be used to + parallelize tests over more machines. + +## Release 2.0.4 (2020-02-21) + +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the versions currently used on LGTM.com. These are newer than the + last release of LGTM Enterprise, so this release should not be used + if you plan to upload databases to an LGTM Enterprise instance. For + more information, see [Preparing CodeQL databases to upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. + +### Features added + +- Subcommands that execute queries (such as `codeql database analyze`) + now have a `--timeout` option that can be used to set a timeout to + automatically cancel query evaluations that appear to diverge. +- A new plumbing command `codeql query decompile` can display the DIL + intermediate representations that is included in the output of + `codeql query compile --dump-qlo --include-dil-in-qlo`. This is + useful mainly for certain internal workflows; the information + produced is the same as what `codeql query compile --dump-dil` + already outputs. + +### Bugs fixed + +- The `--debug` and `--tuple-counting` options to + `codeql test run` erroneously had no effect. Now they ought to work. + +## Release 2.0.3 (2020-02-12) + +### Bugs fixed + +- Fixes a bug where `codeql test run` would fail with the + message + `CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it` + when running tests against the `master` branch of the CodeQL + libraries for certain languages. +- Otherwise identical to release 2.0.2. + +## Release 2.0.2 (2020-02-05) + +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the versions currently used on LGTM.com. These are newer than the + last release of LGTM Enterprise, so this release should not be used + if you plan to upload databases to an LGTM Enterprise instance. For + more information, see [Preparing CodeQL databases to upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. +- The parent and sibling directories of the unpacked CLI are no longer + searched recursively for QL packs. QL packs will only be found if + there's a `qlpack.yml` or `.codeqlmanifest.json` directly in a + parent or sibling directory. This should eliminate the very long + disk-scanning delays experienced by users who unpacked earlier + versions of the CLI in their home directory. +- Parent and sibling directories of the unpacked CLI will now be + searched for QL packs as a last resort, even if you give an explicit + `--search-path` option. This means, for example, that you can define + a search path in the [per-user configuration file](https://help.semmle.com/codeql/codeql-cli/reference/configuration-overview.html) without it depending on + where the CLI is unpacked. In particular, the setting can now be + meaningfully used by users who let the CodeQL for VS Code extension + manage the downloading and unpacking of the CLI. + +### Security updates + +- The `codeql database create` command and its relatives will no + longer attempt to find extractors located in the parent and sibling + directories of the unpacked CLI. This closes a security risk for + users who unpacked the CodeQL CLI in their home directory. This + could've resulted in arbitrary code execution if the user unpacked a + file archive containing a malicious extractor anywhere in the home + directory. Extractors will now only be found within the unpacked CLI + itself, or in directories explicitly listed in the `--search-path`. + It is expected that users will only point `--search-path` to + locations they trust at least as much as the CLI download itself. + +### Features added + +- This release supports executing query regression tests using the + `codeql test` command. For further information, see + [Testing custom queries](https://help.semmle.com/codeql/codeql-cli/procedures/test-queries.html). +- The error message if you try executing a query against a database + that needs to be upgraded (which can happen routinely if you're + using a fresh `master` checkout of the CodeQL libraries with the + bundled extractors) will now explicitly suggest a + `codeql database update` command to run. The database is not + automatically upgraded, as this may make it irreversibly + incompatible with older versions of the CodeQL libraries. This + allows users who want to compare behavior of different versions of + the libraries against the same database to make a copy before they + upgrade it. + +## Release 2.0.1 (2019-12-17) + +- Corresponds to LGTM Enterprise release 1.23. +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the extractor versions used in LGTM Enterprise. +- No other changes to the core CLI. + +## Release 2.0.0 (2019-11-14) + +- First public release. + From 7b565e9dc7c0b7bc0ca2d6361bcc298129dc86b4 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 29 Jun 2020 14:03:20 +0200 Subject: [PATCH 007/196] Release 2.2.4 --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f68af8b..68df744 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,28 @@ # CodeQL CLI changelog +## Release 2.2.4 (2020-06-29) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.24) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.24 instance, you need to create them with release + 2.1.4. + +### Bugs fixed + +- QL packs found through the `--search-path` option, or in a sibling + directory to the unpacked CLI would erroneously take precedence over + the content of the workspace when using the CodeQL extension for + Visual Studio Code. This is now fixed such that the workspace + takes priority. + +- Two command-line options that control the amount of disk space that + the QL evaluator will try to keep free of disk cache are now called + `--min-disk-free` and `--min-disk-free-pct`. Previously they were + called `--max-disk-free` instead, which made no sense. The old names + are still recognized such as not to break existing scripts, but are + now undocumented and deprecated. + ## Release 2.2.3 (2020-06-15) CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new From c68c82bb7f5e9ea7c948d3361c09d7e34d3b9a04 Mon Sep 17 00:00:00 2001 From: Aditya Sharad <6874315+adityasharad@users.noreply.github.com> Date: Thu, 30 Jul 2020 17:13:40 -0700 Subject: [PATCH 008/196] Actions: Autolabel issues when opened --- .github/workflows/label-issue.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 .github/workflows/label-issue.yml diff --git a/.github/workflows/label-issue.yml b/.github/workflows/label-issue.yml new file mode 100644 index 0000000..9c2567c --- /dev/null +++ b/.github/workflows/label-issue.yml @@ -0,0 +1,16 @@ +name: Label issue +on: + issues: + types: [opened] + +jobs: + label: + name: Label issue + runs-on: ubuntu-latest + if: github.event.action == 'opened' + steps: + - name: Label issue + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + echo '{"labels": ["CLI"]}' | gh api repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/labels --input - From 1e49f2d4c29caab5c3f3a14f3b6d910e99176264 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 21 Aug 2020 13:52:47 +0200 Subject: [PATCH 009/196] changelog for 2.2.5 --- CHANGELOG.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 68df744..a694232 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ # CodeQL CLI changelog +## Release 2.2.5 (2020-08-21) + + - The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.24) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.24 instance, you need to create them with release + 2.1.4. + + - Updated license terms with a rewritten description of what is and + is not allowed. No substantive changes are intended, but the new + text is hopefully easier to understand. + +### New features + + - The CLI can now execute queries that use QL's `external predicate` + feature. All subcommands that execute queries have a new + `--external` option to specify the value set for those predicates. + + - A new `codeql bqrs diff` command can be used to compute the + difference between two binary query result sets. + + - `codeql test run` has some new options to improve support for + testing of extractors: + - `--check-databases` which will run `codeql dataset check` on + every test database produced during a run. + - `--consistency-queries` which will run a set of additional + queries over _all_ the test databases produced during a run. + - `--show-extractor-output` + ## Release 2.2.4 (2020-06-29) - The bundled extractors are updated to match the versions currently From 66a113954d0b73133ad570f1473adad3201168fb Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 21 Aug 2020 13:54:52 +0200 Subject: [PATCH 010/196] update license --- LICENSE.md | 96 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 63 insertions(+), 33 deletions(-) diff --git a/LICENSE.md b/LICENSE.md index 3616ada..3ef032d 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -4,52 +4,81 @@ These GitHub CodeQL Terms and Conditions ("**Terms**") are a legal agreement between you (either as an individual or on behalf of an entity) and GitHub, Inc. regarding your use of the GitHub CodeQL software and associated documentation (collectively, the -"**Software**"). By using the Software, you accept these Terms. If you -do not accept these Terms, do not download, install, use, or copy the -Software. +"**Software**"). By using the Software, you accept these Terms. +**Please read all of these Terms;** in many cases, provisions set +forth later in the Terms limit and qualify provisions set forth +earlier in the Terms. If you do not accept these Terms, do not +download, install, use, or copy the Software. + +Definitions +----------- + +In these Terms: + + * "OSI-approved License" means an Open Source Initiative + (OSI)-approved open source software license. + + * "Open Source Codebase" means a codebase that is released under an + OSI-approved License. Use Rights; Scope of License ---------------------------- -The Software is licensed on a per user basis. You may use the Software -for any of the following: +The Software is licensed on a per user basis. Here's what you may do +with the Software, but subject to License Restrictions provisions +below: - * To perform analysis on any codebase that is released under an Open - Source Initiative (OSI)-approved open source software license (an - “Open Source Codebase”); + * Use the Software to perform academic research. - * To generate CodeQL databases for or during automated analysis, - continuous integration (CI) or continuous delivery (CD) in the - following cases: (1) on any Open Source Codebase hosted and - maintained on GitHub.com, and (2) to test CodeQL queries you have - released under an OSI-approved open source software license to - confirm that new versions of those queries continue to find the - right vulnerabilities; + * Use the Software to demonstrate the Software. - * To perform academic research on any codebase; and + * Test CodeQL queries that are released under an OSI-approved + Licence to confirm that new versions of those queries continue to + find the right vulnerabilities. - * To demonstrate the Software. +Here's what you may also do with the Software, but only with an Open +Source Codebase and subject to the License Restrictions provisions +below: + + * Perform analysis on the Open Source Codebase. + + * If the Open Source Codebase is hosted and maintained on + GitHub.com, generate CodeQL databases for or during automated + analysis, CI, or CD. + +License Restrictions +-------------------- -**These Terms do not authorize, and the Software cannot be used, for -any purpose not expressly set forth above, including to generate any -CodeQL database for or during automated analysis, CI or CD, whether as -part of normal engineering processes or otherwise, except in the -express cases set forth above.** +These Terms do not authorize, and the Software may not be used for any +purpose not expressly set forth above, including: -Further, except (and only to the extent) permitted by applicable law -or applicable third-party license, you will not (and have no right + * To otherwise or in any other context generate any CodeQL database + for or during automated analysis, CI or CD, whether as part of + normal engineering processes or another context. + + * To otherwise or in any other context use the Software in + connection with any codebase that is not an Open Source Codebase + (e.g., code in a private repo in GitHub). + +_**Please note:** if your use of the Software is under a paid customer +license for GitHub Advanced Security, the restrictions with respect to +automated analysis, CI, and CD and use in connection with non-Open +Source Codebases do not apply._ + +At all times, except (and only to the extent) permitted by applicable +law or applicable third-party license, you will not (and have no right to): * work around any technical limitations in the Software that only allow you to use it in certain ways; * reverse engineer, decompile or disassemble the Software; - + * remove, minimize, block, or modify any notices of GitHub or its suppliers in the Software; - + * use the Software in any way that is against the law; or - + * share, publish, distribute or lend the Software, provide or make available the Software as a hosted solution (whether on a standalone basis or combined, incorporated or integrated with @@ -144,15 +173,16 @@ software licenses referenced above, constitutes the entire agreement between you and GitHub regarding your use of the Software, superseding any prior agreements between you and GitHub (including, but not limited to, any prior versions of these Terms) regarding such use. -Governing Law. You agree that these Terms and your use of the Software -are governed by the laws of the State of California and any dispute -relating to the Software or your use thereof must be brought in a -tribunal of competent jurisdiction located in or near San Francisco, -California. + +_Governing Law._ You agree that these Terms and your use of the +Software are governed by the laws of the State of California and any +dispute relating to the Software or your use thereof must be brought +in a tribunal of competent jurisdiction located in or near San +Francisco, California. _Modifications._ These Terms may only be modified by a written amendment signed by an authorized representative of GitHub, or by the posting by GitHub of a revised version. _Contact Us._ Questions about these Terms? Contact us at -https://support.github.com/contact. \ No newline at end of file +https://support.github.com/contact. From 9f8dd71e3aad0fe3afb34f61f9e69a628057081f Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Wed, 9 Sep 2020 19:06:54 +0200 Subject: [PATCH 011/196] Release 2.2.6 --- CHANGELOG.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a694232..b84f178 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # CodeQL CLI changelog +## Release 2.2.6 (2020-09-11) + +This release corresponds to release 1.25.x of LGTM Enterprise, and +should be used when creating databases that will be uploaded to it. +Future CLI releases (numbered 2.3.x) may produce databases that are not +backwards compatible with this version of LGTM Enterprise. + +For all purposes other than creating databases for LGTM Enterprise we +recommend that you continue upgrading to newer CLI releases as they +become available. + ## Release 2.2.5 (2020-08-21) - The bundled extractors are updated to match the versions currently @@ -410,4 +421,3 @@ become available. ## Release 2.0.0 (2019-11-14) - First public release. - From 77175f907f2c1116bf0acf3171f8e5a37dc965cf Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 28 Sep 2020 19:44:53 +0200 Subject: [PATCH 012/196] changelog for 2.3.0 --- CHANGELOG.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b84f178..6921a15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,49 @@ # CodeQL CLI changelog +## Release 2.3.0 (2020-09-28) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.25) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.25 instance, you need to create them with release + 2.2.6. + +### Potentially breaking changes + +- The Java extractor no longer supports builds running on a Java 6 + JRE. The minimum supported version is Java 7. + +- The interpretation of binding set annotations in QL has changed + subtly. In rare cases, existing QL code that contains explicit + binding set annotations on overriding class predicates may now be + rejected with errors of the form "... is not bound to a value". You + can fix this by adding explicit binding sets to the overridden + predicate, or to the abstract class itself in the case of the + characteristic predicate. For more information about binding sets, + see [Annotations](https://help.semmle.com/QL/ql-handbook/annotations.html#binding-sets) + in the QL language reference. + +### QL language improvements + +- You can now use binding sets on class bodies. This lets you + explicitly annotate dynamically dispatched characteristic + predicates. + +### New features + +- Query authors can use the new subcommand `codeql generate query-help` to + validate query help files and render the files as Markdown. For more information, + see [Testing query help files](https://help.semmle.com/codeql/codeql-cli/procedures/testing-query-help-files.html). + +- The new subcommand `codeql bqrs hash` computes a stable hash of a + BQRS file. + +- `codeql query decompile` now accepts a `--kind` flag. This allows + advanced users to choose which intermediate representation to show + for a compiled QL query. `--kind dil` shows the Datalog + representation while `--kind ra` shows the relational algebra + representation used by the evaluator. + ## Release 2.2.6 (2020-09-11) This release corresponds to release 1.25.x of LGTM Enterprise, and From 7ffdadaef685900e7a795a351ac8c0fc9da26d94 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 15 Oct 2020 17:12:43 +0200 Subject: [PATCH 013/196] changelog for 2.3.1 --- CHANGELOG.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6921a15..717d2b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # CodeQL CLI changelog +## Release 2.3.1 (2020-10-15) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.25) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.25 instance, you need to create them with release + 2.2.6. + +### Features added + +- `codeql database create` now accepts a `--working-dir` option, which + allows the working directory for extractor scripts to differ from + the source root. This is useful in some specialized integration + situations. + +- `codeql database create` will now pass a `--compiler-spec` option on + to `codeql database trace-command`. This allows adapting the build + tracing process when unusual compiler toolchains are used. + +- `codeql database init` accepts an `--allow-missing-source-root` + option, which is useful in some specialized integration situations. + ## Release 2.3.0 (2020-09-28) - The bundled extractors are updated to match the versions currently From 682f40d900b2a55045fe4fd6a18083d8c26a0cb6 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Tue, 27 Oct 2020 19:42:04 +0100 Subject: [PATCH 014/196] changelog for 2.3.2 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 717d2b0..ad8367c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # CodeQL CLI changelog +## Release 2.3.2 (2020-10-27) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.25) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.25 instance, you need to create them with release + 2.2.6. + ## Release 2.3.1 (2020-10-15) - The bundled extractors are updated to match the versions currently From 46bcb1464f1c03952fbc836d04ca9d9daad13480 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 6 Nov 2020 20:16:48 +0100 Subject: [PATCH 015/196] changelog for 2.3.3 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad8367c..6abb965 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,22 @@ # CodeQL CLI changelog +## Release 2.3.3 (2020-11-06) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.25) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.25 instance, you need to create them with release + 2.2.6. + +- Fixed bug in `codeql test run` where the `--json` and + `--failing-exitcode` options would not work reliably when `--ram` + was also given + +- The `$CODEQL_JAVA_HOME` environment variable will now be passed to + extractors such that extractors implemented in Java can be affected + too. Beware that this variable will override the JVM that executes + the main `codeql` process. It should not normally be set explicitly. + ## Release 2.3.2 (2020-10-27) - The bundled extractors are updated to match the versions currently From caf18cb95b43fc716f9b95c36cf2b8579093589b Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 12 Nov 2020 19:47:04 +0100 Subject: [PATCH 016/196] fix wrong option name --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6abb965..37d25fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ Enterprise 1.25 instance, you need to create them with release 2.2.6. -- Fixed bug in `codeql test run` where the `--json` and +- Fixed bug in `codeql test run` where the `--format` and `--failing-exitcode` options would not work reliably when `--ram` was also given From 6ac95de2e69cc79cc77aff2958fc40a97462ea8e Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 26 Nov 2020 17:26:21 +0100 Subject: [PATCH 017/196] changelog for 2.4.0 --- CHANGELOG.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 37d25fc..0c441a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # CodeQL CLI changelog +## Release 2.4.0 (2020-11-25) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.25) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.25 instance, you need to create them with release + 2.2.6. + +- Much of the work done by `codeql database upgrade` now happens + implicitly (and reversibly) as part of ordinary query evaluation. + This should make it much rarer to need to run `codeql database + upgrade` explicitly, though there are still some corner cases that + will require it, particularly for very old databases. + +- `codeql test run` with a `--threads` argument will now _compile_ + test queries in parallel even if they belong to the same single + test directory. This can speed up localized testing considerably. + ## Release 2.3.3 (2020-11-06) - The bundled extractors are updated to match the versions currently From 88c614c3c81d63f9f88f797ad1c11839160b487d Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Wed, 16 Dec 2020 00:09:51 +0100 Subject: [PATCH 018/196] changelog for 2.3.4 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c441a3..3b3550e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # CodeQL CLI changelog +## Release 2.3.4 (2020-12-15) + +This release corresponds to release 1.26.x of LGTM Enterprise, and +should be used when creating databases that will be uploaded to it. +Future CLI releases (numbered 2.4.x) may produce databases that are not +backwards compatible with this version of LGTM Enterprise. + +For all purposes other than creating databases for LGTM Enterprise we +recommend that you upgrade to CLI releases numbered 2.4.x or later. + ## Release 2.4.0 (2020-11-25) - The bundled extractors are updated to match the versions currently From 85ede89cc45cd47a257760dae631d406213a196c Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Sat, 19 Dec 2020 02:44:07 +0100 Subject: [PATCH 019/196] changelog for 2.4.1 --- CHANGELOG.md | 41 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b3550e..299c8d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,40 @@ # CodeQL CLI changelog +## Release 2.4.1 (2020-12-19) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.26) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.26 instance, you need to create them with release + 2.3.4. + +### Features added + +- `codeql query format` now checks all files rather than stopping + after the first failure when the `--check-only` option is given. + +- `codeql resolve database` will produce a `languages` key giving the + language the database was created for. This can be useful in IDEs to + help describe the database and suggest default actions or queries. + For databases created by earlier versions, the result will be a + best-effort guess. + +- `codeql database interpret-results` can now produce Graphviz `.dot` + files from queries with `@kind graph`. + +### Features removed + +- `codeql test run` had some special compatibility support for running + unit tests for the "code duplication" extractor features of certain + discontinued Semmle products. Those tests have since been removed + from the [public QL repository](https://github.com/github/codeql), + so the compatibility support for them has been removed. This should + not affect any external users (since the extractor feature in + question was never supported by `codeql database create` anyway), + but if you run `codeql test run` against the unit tests belonging to + an _old_ checkout of the repository, you may now see some failures + among `Metrics` tests. + ## Release 2.3.4 (2020-12-15) This release corresponds to release 1.26.x of LGTM Enterprise, and @@ -20,9 +55,9 @@ recommend that you upgrade to CLI releases numbered 2.4.x or later. - Much of the work done by `codeql database upgrade` now happens implicitly (and reversibly) as part of ordinary query evaluation. - This should make it much rarer to need to run `codeql database - upgrade` explicitly, though there are still some corner cases that - will require it, particularly for very old databases. + This should make the need to explicitly run `codeql database + upgrade` much less common. However there are still some corner cases + that will require it, particularly for very old databases. - `codeql test run` with a `--threads` argument will now _compile_ test queries in parallel even if they belong to the same single From 6f0f178f610e941f2e3cdace025b4adb3d1551d7 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 22 Jan 2021 20:37:57 +0100 Subject: [PATCH 020/196] changelog for 2.4.1 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 299c8d2..2b2b91d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # CodeQL CLI changelog +## Release 2.4.2 (2021-01-22) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.26) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.26 instance, you need to create them with release + 2.3.4. + ## Release 2.4.1 (2020-12-19) - The bundled extractors are updated to match the versions currently From f46210739119bdf2c8c3b37ba928e62ce5972527 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 29 Jan 2021 23:59:59 +0100 Subject: [PATCH 021/196] changelog for 2.4.3 --- CHANGELOG.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b2b91d..919ddae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # CodeQL CLI changelog +## Release 2.4.3 (2021-01-29) + +Fixes several bugs introduced in 2.4.2, related to searching the disk for +QL packs: + +- In many cases the search would scan through more of the file system + than it should. Often the only effect of this was that the scan would + take longer time (sometimes significantly longer) but in some corner + cases it could lead to packs being found that _shouldn't_ be found, + which could lead to compilation failure if different versions of the same + pack exists on disk. + +- The search would terminate a fatal error if it met a directory without + read permission. + +- A `provide` entry in `.codeqlmanifest.json` that ended with `*` would + erroneously not match a `.codeqlmanifest.json` in a subdirectory. + +As a consequence of the latter fix, the semantics of +`.codeqlmanifest.json` files has changed slightly: Directory names +that start with a dot used not to be matched by the pattern elements +`*` and `**`, whereas now even dotted directories match such a pattern +element. The previous behavior was never documented, and only very few +users have `.codeqlmanifest.json` files of their own in the first +place, so this change is expected to have minimal practical effect. + ## Release 2.4.2 (2021-01-22) - The bundled extractors are updated to match the versions currently From 5ae9698ebf76955d0b1f81471c147ef69fc6cc06 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 12 Feb 2021 21:37:22 +0100 Subject: [PATCH 022/196] changelog for 2.4.4 --- CHANGELOG.md | 45 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 919ddae..aabd8f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,44 @@ # CodeQL CLI changelog +## Release 2.4.4 (2021-02-12) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.26) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.26 instance, you need to create them with release + 2.3.4. + +### Potentially breaking changes + +- The `name` property in `qlpack.yml` must now meet the following requirements: + - Only lowercase ASCII letters, ASCII digits, and hyphens (`-`) are allowed. + - A hyphen is not allowed as the first or last character of the name. + - The name must be at least one character long, and no longer than 128 characters. + +### New features + +- Alert and path queries can now give a score to each alert they + produce. You can incorporate alert scores in an alert or path query + by first adding the `@scored` property to the query metadata. You + can then introduce a new numeric column at the end of the `select` + statement structure to represent the score of each alert. + Alert scores are exposed in the SARIF output of commands like + `codeql database analyze` as the `score` property in the property + bags of result objects. + +### Bugs fixed + +- The default value of the `--working-dir` options for the + `index-files` and `trace-command` subcommands of `codeql database` + has been fixed to match the documentation; previously, it would + erroneously use the process' current working directory rather than + the database source root. + +- `codeql test run` will not crash if database extraction in a test + directory fails. Instead only the tests in that directory will be + marked as failing, and tests in other directories will continue + executing. + ## Release 2.4.3 (2021-01-29) Fixes several bugs introduced in 2.4.2, related to searching the disk for @@ -7,10 +46,10 @@ QL packs: - In many cases the search would scan through more of the file system than it should. Often the only effect of this was that the scan would - take longer time (sometimes significantly longer) but in some corner + take longer (sometimes significantly longer) but in some corner cases it could lead to packs being found that _shouldn't_ be found, which could lead to compilation failure if different versions of the same - pack exists on disk. + pack exist on disk. - The search would terminate a fatal error if it met a directory without read permission. @@ -20,7 +59,7 @@ QL packs: As a consequence of the latter fix, the semantics of `.codeqlmanifest.json` files has changed slightly: Directory names -that start with a dot used not to be matched by the pattern elements +that start with a dot used to not be matched by the pattern elements `*` and `**`, whereas now even dotted directories match such a pattern element. The previous behavior was never documented, and only very few users have `.codeqlmanifest.json` files of their own in the first From d64d8fa6d6b9e457a49d7b30f3ce2664d09d132f Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 8 Mar 2021 20:51:35 +0100 Subject: [PATCH 023/196] changelog for 2.4.5 --- CHANGELOG.md | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aabd8f1..daff1f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,26 @@ # CodeQL CLI changelog +## Release 2.4.5 (2021-03-08) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.26) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.26 instance, you need to create them with release + 2.3.4. + +- The C/C++ extractor can now parse more Microsoft language extensions when in + C++14 and C++17 mode. + +- `codeql database analyze` now reports the name and version of each + QL pack used by the analysis. You can find this information in the + SARIF output. In particular, the `runs[0].tool.extensions` property + contains an object for each QL pack used by the analysis. Each + object contains the `name` and `semanticVersion` of the + corresponding QL pack, if such information is available. + +- `codeql github upload-sarif` is a new command that uploads a SARIF file + generated by CodeQL to GitHub's Code Scanning. + ## Release 2.4.4 (2021-02-12) - The bundled extractors are updated to match the versions currently @@ -214,8 +235,8 @@ recommend that you upgrade to CLI releases numbered 2.4.x or later. ### New features -- Query authors can use the new subcommand `codeql generate query-help` to - validate query help files and render the files as Markdown. For more information, +- Query authors can use the new subcommand `codeql generate query-help` to + validate query help files and render the files as Markdown. For more information, see [Testing query help files](https://help.semmle.com/codeql/codeql-cli/procedures/testing-query-help-files.html). - The new subcommand `codeql bqrs hash` computes a stable hash of a @@ -279,7 +300,7 @@ become available. - QL packs found through the `--search-path` option, or in a sibling directory to the unpacked CLI would erroneously take precedence over - the content of the workspace when using the CodeQL extension for + the content of the workspace when using the CodeQL extension for Visual Studio Code. This is now fixed such that the workspace takes priority. @@ -363,7 +384,7 @@ become available. ## Release 2.1.3 (2020-05-13) -### Bugs fixed +### Bugs fixed - Fixes a bug in `codeql execute cli-server` (a helper used by the VS Code extension) which would sometimes cause query compilation to From 0cc3237f97596d0b94204ef471bf6026841d1595 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Wed, 10 Mar 2021 15:50:35 +0000 Subject: [PATCH 024/196] Fix incorrect option name in changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index daff1f9..0457321 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,7 @@ object contains the `name` and `semanticVersion` of the corresponding QL pack, if such information is available. -- `codeql github upload-sarif` is a new command that uploads a SARIF file +- `codeql github upload-results` is a new command that uploads a SARIF file generated by CodeQL to GitHub's Code Scanning. ## Release 2.4.4 (2021-02-12) From 1264f30a1036b5e6e5ed9dddf58e90f817f6ec1b Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Sat, 20 Mar 2021 00:05:26 +0100 Subject: [PATCH 025/196] changelog for 2.4.6 --- CHANGELOG.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0457321..e4b67b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # CodeQL CLI changelog +## Release 2.4.6 (2021-03-19) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.26) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.26 instance, you need to create them with release + 2.3.4. + +- Fixed a bug in `codeql test run` that causes tests to fail messily + if the freshly-extracted test database needed to be upgraded in + order to be compatible with the QL source under test. This would + happen more often at the _end_ of a release cycle, after updates to + the QL repository had happened. + +- `codeql github upload-results` should now work correctly against + GitHub Enterprise Server instances that are configured with a path + prefix. + ## Release 2.4.5 (2021-03-08) - The bundled extractors are updated to match the versions currently From e813a54e6d2836a0865ad1fed1f194a2631257c5 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Wed, 24 Mar 2021 23:47:57 +0100 Subject: [PATCH 026/196] 2.4.6 corresponds to LGTM-e 1.27 --- CHANGELOG.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4b67b7..b699f3e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,10 @@ ## Release 2.4.6 (2021-03-19) -- The bundled extractors are updated to match the versions currently - used on LGTM.com. These are newer than the last release (1.26) of - LGTM Enterprise. If you plan to upload databases to an LGTM - Enterprise 1.26 instance, you need to create them with release - 2.3.4. +This release corresponds to release 1.27.x of LGTM Enterprise, and +should be used when creating databases that will be uploaded to it. +Future CLI releases (numbered 2.5.x) may produce databases that are not +backwards compatible with this version of LGTM Enterprise. - Fixed a bug in `codeql test run` that causes tests to fail messily if the freshly-extracted test database needed to be upgraded in From 42d0cdf1f6227cfa3eb6241212fe0b02aa406afe Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 26 Mar 2021 14:46:39 +0100 Subject: [PATCH 027/196] changelog for 2.5.0 --- CHANGELOG.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b699f3e..7bafa1b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,37 @@ # CodeQL CLI changelog +## Release 2.5.0 (2021-03-29) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Potentially breaking changes + +- By default, `codeql test` now performs additional compiler + checks when extracting test code written in Java. + Existing Java tests that previously passed may therefore fail due + to this change, if they do not compile using the `javac` compiler. + To allow time to migrate existing tests, the new behavior can be + disabled by setting the environment variable + `CODEQL_EXTRACTOR_JAVA_FLOW_CHECKS=false`. + +### Features added + +- Log files that contain output from build processes will now prefix + it with `[build-stdout]` and `[build-stderr]` instead of `[build]` + and `[build-err]`. In particular the latter sometimes caused + confusion. + +### QL language improvements + +- The QL language now recognizes new `pragma[only_bind_into](...)` and + `pragma[only_bind_out](...)` annotations on expressions. Advanced users + may use these annotations to provide hints to the compiler to influence + binding behavior and thus indirectly performance. + ## Release 2.4.6 (2021-03-19) This release corresponds to release 1.27.x of LGTM Enterprise, and From 7765dc882b0c91ec471ea0f8b901aa5c180efb93 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 26 Mar 2021 15:20:28 +0100 Subject: [PATCH 028/196] correct date for 2.5.0 release --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bafa1b..39da40a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # CodeQL CLI changelog -## Release 2.5.0 (2021-03-29) +## Release 2.5.0 (2021-03-26) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of From bac6f8e9f54da1173282d35bcfe62fba37ef0284 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 19 Apr 2021 18:46:49 +0200 Subject: [PATCH 029/196] changelog for release 2.5.1 --- CHANGELOG.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 39da40a..8b3cb8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,62 @@ # CodeQL CLI changelog +## Release 2.5.1 (2021-04-19) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Potentially breaking changes + +- The QL compiler will now reject queries where the query metadata (if + present) at the top of the `.ql` file is inconsistent with the + output format of the query. This check can be disabled by giving + the `--no-metadata-verification` flag. (The flag already existed + but has not had any effect until now.) + +### Bugs fixed + +- Environment variables required for Java extraction are now + propagated by the tracer. This may resolve issues with tracing and + extraction in the context of certain build systems such as Bazel. + +- A number of `--check-CONDITION` options to `codeql database + finalize` and `codeql dataset import` designed to look for + consistency errors in the intermediate "TRAP" output from extractors + erroneously did nothing. They will now actually print warnings if + errors are found. The warnings become fatal errors if the new + `--fail-on-trap-errors` option is also given. + +### Features added + +- `codeql resolve qlref` is a new command that takes in a `.qlref` + file for a CodeQL test case and returns the path of the `.ql` file + it references. + +- `codeql database analyze` and `codeql database interpret-results` + have a new `--sarif-group-rules-by-pack` option which will place the + SARIF rule object for each query underneath its corresponding query + pack in `runs[].tool.extensions`. + +- `codeql database finalize` and `codeql dataset import` have a new + `--fail-on-trap-errors` option that will make database creation fail + if extractors produce ill-formatted "TRAP" data for inclusion into a + database. This is not enabled by default because some of the + existing extractors have minor output bugs that cause the check to + fail. + +- `codeql database finalize` and `codeql dataset import` have a new + `--check-undefined-labels` option that enables stricter consistency + checks on the "TRAP" output from extractors. + +### QL language improvements + +- `super` may now be used unqualified, e.g. `super.predicateName()`, + when the declaring class has multiple super types, as long as the + call itself is unambiguous. + ## Release 2.5.0 (2021-03-26) - The bundled extractors are updated to match the versions currently From a97180a72b66e4fcfc531cc45e22bc85b461a92e Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Wed, 21 Apr 2021 18:09:32 +0200 Subject: [PATCH 030/196] changelog for 2.5.2 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b3cb8c..8b805ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # CodeQL CLI changelog +## Release 2.5.2 (2021-04-21) + +This release is identical to release 2.5.1, except that an internal +incompatibility with the CodeQL action (and the codeql-runner that +some customers use for CI integrations) has been fixed. + +The fix does not affect any use cases where the CLI is downloaded from +github/codeql-cli-binaries, so if you're seeing this release there, +there's no need to upgrade from 2.5.1. + ## Release 2.5.1 (2021-04-19) - The bundled extractors are updated to match the versions currently From 57efe2521d47a66ce76c12889c00d96922300bb9 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 29 Apr 2021 20:01:03 +0200 Subject: [PATCH 031/196] changelog for release 2.5.3 --- CHANGELOG.md | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b805ba..0015d72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,66 @@ # CodeQL CLI changelog +<-- ********************************* + ** ** + ** DO NOT EDIT THIS FILE! ** + ** ** + ********************************* +. + Pull requests should instead drop Markdown snippets in the + subdirectory `unreleased-changelog-entries` found next to + the authoritative copy of this file in semmle-code. + Please include a three-hash heading such as "Bugs Fixed" + and format your snippet as a list item. +. + (Okay, if you're the CLI release manager following the + checklist for a CLI release, you can edit here. But then + you know what to do). +--> + +## Release 2.5.3 (2021-04-30) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Features added + +- `codeql database analyze` and `codeql database interpret-results` + now report the results of summary metric queries in the + `.properties.metricResults` property of the SARIF output. + Summary metric queries describe metrics about the code analyzed by + CodeQL. They are identified by the query metadata `@kind metric` and + `@tag summary`. + For example, see the [lines of code summary metric query for + C++](https://github.com/github/codeql/blob/main/cpp/ql/src/Summary/LinesOfCode.ql). + +- `codeql database analyze` and `codeql database interpret-results` + now calculate an + [automation ID](https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/sarif-v2.1.0-cs01.html#_Toc16012482) + and add it to the resulting SARIF. In SARIF v2.1.0, this field is + `runs[].automationDetails.id`. In SARIF v2, this field is + `runs[].automationLogicalId`. In SARIF v1, this field is + `runs[].automationId`. By default, this automation ID will be + derived from the database language and the operating system of the + machine that performed the run. It can be set explicitly using a new + `--sarif-category` option. + +- In query metadata, `@kind alert` and `@kind path-alert` are now + recognized as (more accurate) synonyms of `@kind problem` and + `@kind path-problem`, respectively. + +- Diagnostic queries are now permitted by the metadata verifier. They + are identified by `@kind diagnostic` metadata. Currently the result + patterns of diagnostic queries are not verified. This will change in + a future CLI release. + +### Bugs fixed + +- Ensure the correct URL is generated during `codeql github upload-results` + for GitHub Enterprise Server. + ## Release 2.5.2 (2021-04-21) This release is identical to release 2.5.1, except that an internal From f675698a6d516638b8869f5ee6d87f0a8d2ef46c Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 30 Apr 2021 18:49:51 +0200 Subject: [PATCH 032/196] hmm, the HTML commend didn't disappear in rendering ... --- CHANGELOG.md | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0015d72..0ec8f50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,22 +1,5 @@ # CodeQL CLI changelog -<-- ********************************* - ** ** - ** DO NOT EDIT THIS FILE! ** - ** ** - ********************************* -. - Pull requests should instead drop Markdown snippets in the - subdirectory `unreleased-changelog-entries` found next to - the authoritative copy of this file in semmle-code. - Please include a three-hash heading such as "Bugs Fixed" - and format your snippet as a list item. -. - (Okay, if you're the CLI release manager following the - checklist for a CLI release, you can edit here. But then - you know what to do). ---> - ## Release 2.5.3 (2021-04-30) - The bundled extractors are updated to match the versions currently From 0e8bef188e7d65534954213c20ba8292b14e0b18 Mon Sep 17 00:00:00 2001 From: Aditya Sharad <6874315+adityasharad@users.noreply.github.com> Date: Mon, 3 May 2021 15:05:04 -0700 Subject: [PATCH 033/196] CHANGELOG: Add 2.5.4 --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ec8f50..2eebac7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # CodeQL CLI changelog +## Release 2.5.4 (2021-05-03) + +- This release is identical to release 2.5.3, except that + `codeql database analyze` no longer produces a generated + `automationDetails.id` field when the `--sarif-category` + is not explicitly provided. Previously, the + `--sarif-category` was autogenerated if not present. +- Code Scanning users should upgrade to this version and + avoid 2.5.3. +- If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + ## Release 2.5.3 (2021-04-30) - The bundled extractors are updated to match the versions currently From f00e93e35bb0430192c54a8e6991d029a08f2341 Mon Sep 17 00:00:00 2001 From: Marcono1234 Date: Tue, 4 May 2021 11:43:24 +0200 Subject: [PATCH 034/196] Change Semmle to GitHub links in README --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 61acc2a..4617044 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ This repo holds binaries for the CodeQL CLI. -[CodeQL overview](https://securitylab.github.com/tools/codeql) | [CodeQL CLI Documentation](https://help.semmle.com/codeql/codeql-cli.html) +[CodeQL overview](https://securitylab.github.com/tools/codeql/) | [CodeQL CLI Documentation](https://codeql.github.com/docs/codeql-cli/) # Getting started @@ -10,12 +10,12 @@ This repo holds binaries for the CodeQL CLI. 2. Find the latest release, select Assets, and download the zip file containing the CLI. 3. You'll also want to clone https://github.com/github/codeql to get the CodeQL queries and libraries. Please take note of the - [set-up instructions](https://help.semmle.com/codeql/codeql-cli/procedures/get-started.html) + [set-up instructions](https://codeql.github.com/docs/codeql-cli/getting-started-with-the-codeql-cli/) for placing it in a location where the CLI can find it. -4. Read the rest of the [CodeQL CLI documentation](https://help.semmle.com/codeql/codeql-cli.html). +4. Read the rest of the [CodeQL CLI documentation](https://codeql.github.com/docs/codeql-cli/). # License -By downloading, you agree to the [GitHub CodeQL Terms & Conditions](https://securitylab.github.com/tools/codeql/license). +By downloading, you agree to the [GitHub CodeQL Terms & Conditions](https://securitylab.github.com/tools/codeql/license/). GitHub CodeQL can only be used on codebases that are released under an OSI-approved open source license, or to perform academic research. It can't be used to generate CodeQL databases for or during automated analysis, continuous integration or continuous delivery, whether as part of normal software engineering processes or otherwise. For these uses, [contact the sales team](https://enterprise.github.com/contact). From 769affc296ea6d8caa616df33029dcb3afc6ec41 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 17 May 2021 21:04:32 +0200 Subject: [PATCH 035/196] changelog for 2.5.5 --- CHANGELOG.md | 70 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 66 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2eebac7..3ae6adf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,70 @@ # CodeQL CLI changelog + + +## Release 2.5.5 (2021-05-17) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Potentially breaking changes + +- When scanning the disk for QL packs and extractors, directories of + the form `.../SOMETHING/SOMETHING.testproj` (where the two + `SOMETHING` are identical) will now be ignored. Names of this form + are used by `codeql test run` for ephemeral test databases, which + can sometimes contain files that confuse QL compilations. + +### Features added + +- Query writers can now optionally use `@severity` in place of + `@problem.severity` in the metadata for alert queries. SARIF + consumers should continue to consume this severity information using + the `rule.defaultConfiguration.level` property for SARIF v2.1.0, and + corresponding properties for other versions of SARIF. They should + not depend on the value stored in the `rule.properties` property + bag, since this will contain either `@problem.severity` or + `@severity` based on exactly what was written in the query metadata. + +- When exporting analysis results to SARIF v2.1.0, results and metric + results now contain a [reporting descriptor reference object][1] + that specifies the rule that produced them. For metric results, this + new property replaces the `metric` property. + + [1]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html#_Toc10541300 + +- `codeql database analyze` now outputs a table that summarises the + results of metric queries that were part of the analysis. This can + be suppressed by passing the `--no-print-metrics-summary` flag. + +### Bugs fixed + +- When using the `--sarif-group-rules-by-pack` flag to place the SARIF + rule object for each query underneath its corresponding query pack + in `runs[].tool.extensions`, the `rule` property of result objects + can now be used to look up the rule within the `rules` property of + the appropriate query pack in `runs[].tool.extensions`. Previously, + rule lookup for result objects in the SARIF output was not + well-defined when the `--sarif-group-rules-by-pack` flag was passed. + ## Release 2.5.4 (2021-05-03) - This release is identical to release 2.5.3, except that @@ -9,10 +74,7 @@ `--sarif-category` was autogenerated if not present. - Code Scanning users should upgrade to this version and avoid 2.5.3. -- If you plan to upload databases to an LGTM - Enterprise 1.27 instance, you need to create them with release - 2.4.6. - + ## Release 2.5.3 (2021-04-30) - The bundled extractors are updated to match the versions currently From 1c2a7acb8656c2d3ead45036262989ca0a6c8ce2 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Tue, 22 Jun 2021 18:39:50 +0200 Subject: [PATCH 036/196] changelog for 2.5.6 --- CHANGELOG.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ae6adf..71cb68f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,67 @@ you know what to do). --> +## Release 2.5.6 (2021-06-22) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Features added + +- `codeql database create` (and the plumbing commands it comprises) + now supports creating databases for a source tree with several + languages while tracing a single build. This is enabled by a new + `--db-cluster` option. Once created, the multiple databases must be + _analyzed_ one by one. + +- `codeql database create` and `codeql database init` now accept an + `--overwrite` argument which will lead existing CodeQL databases to + be overwritten. + +- `codeql database analyze` now supports "diagnostic" queries (tagged + `@kind diagnostic`), which are intended to report information about + the analysis process itself rather than problems with the analyzed + code. The results of these queries will be summarized in a table + printed to the terminal when `codeql database analyze` finishes. + + They are also included in the analysis results in SARIF output + formats as [notification objects][2] so they can be displayed by + subsequent tooling such as the Code Scanning user interface. + + - For SARIF v2.1.0, a reporting descriptor object for each + diagnostic query is output to output to + `runs[].tool.driver.notifications`, or + `runs[].tool.extensions[].notifications` if running with + `--sarif-group-rules-by-pack`. A rule object for each diagnostic + query is output to `runs[].resources[].rules` for SARIF v2, or to + `runs[].rules` for SARIF v1. + + - Results of diagnostic queries are exported to the + `runs[].invocations[].toolExecutionNotifications` property in + SARIF v2.1.0, the `runs[].invocations[].toolNotifications` + property in SARIF v2, and the `runs[].toolNotifications` property + in SARIF v1. + + SARIF v2.1.0 output will now also contain version information for + query packs in `runs[].tool.extensions[].semanticVersion`, if the + Git commit the queries come from is known. + + [2]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html#_Toc34317894 + +- `codeql github upload-results` has a `--checkout-path` option which + will attempt to automatically configure upload target parameters. + When this is given, the `--commit` option will be taken from the + HEAD of the checkout Git repository, and if there is precisely one + remote configured in the local repository, the `--repository` and + `--github-url` options will also be automatically configured. + +- The CodeQL C++ extractor includes beta support for C++20. + This is only available when building codebases with GCC on Linux. + C++20 modules are **not** supported. + ## Release 2.5.5 (2021-05-17) - The bundled extractors are updated to match the versions currently @@ -51,7 +112,7 @@ [1]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html#_Toc10541300 -- `codeql database analyze` now outputs a table that summarises the +- `codeql database analyze` now outputs a table that summarizes the results of metric queries that were part of the analysis. This can be suppressed by passing the `--no-print-metrics-summary` flag. @@ -83,7 +144,12 @@ Enterprise 1.27 instance, you need to create them with release 2.4.6. -### Features added +### Next release: Features added + +- When tracing a C/C++ build, the C compiler entries in compiler-settings + must now specify `order compiler,extractor`. The default configuration + already does this, so no change is necessary if using the default + configuration. - `codeql database analyze` and `codeql database interpret-results` now report the results of summary metric queries in the From 62c16456396d53cc02c47717610b4569ddab46cd Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Fri, 2 Jul 2021 14:44:38 +0200 Subject: [PATCH 037/196] changelog for 2.5.7 --- CHANGELOG.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71cb68f..076f4d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,38 @@ you know what to do). --> +## Release 2.5.7 (2021-07-02) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### New features + +- `codeql database create` and `codeql database init` can now + automatically recognise the languages present in checkouts of GitHub + repositories by making an API call to the GitHub server. This + requires a PAT token to either be set in the `GITHUB_TOKEN` + environment variable, or passed by stdin with the + `--github-auth-stdin` argument. + +- Operations that make outgoing HTTP calls (that is, `codeql github + upload-results` and the language-detection feature described above) + now support the use of HTTP proxies. To use a proxy, specify an + `$https_proxy` environment variable for HTTPS requests or a + `$http_proxy` environment variable for HTTP requests. If the + `$no_proxy` variable is also set, these variables will be ignored + and requests will be made without a proxy. + +### New language features + +- The QL language now has a new method `toUnicode` on the `int` + type. This method converts Unicode codepoint to a one-character string. + For example, `65.toUnicode() = "A"`, `128512.toUnicode()` results in + a smiley, and `any(int i | i.toUnicode() = "A") = 65`. + ## Release 2.5.6 (2021-06-22) - The bundled extractors are updated to match the versions currently From 8add875252111238ec61191968cc9b4b04e04283 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Mon, 26 Jul 2021 18:41:49 +0100 Subject: [PATCH 038/196] Update for 2.5.8 --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 076f4d0..10da2db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,29 @@ you know what to do). --> +## Release 2.5.8 (2021-07-26) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Potentially breaking changes +- The QL compiler now verifies that `@security-severity` query metadata is numeric. You can disable + this verification by passing the `--no-metadata-verification` flag. + +### New features + +- The `database index-files` and `database trace-command` CLI commands now + support `--threads` and `--ram` options, which are passed to extractors as + suggestions. +- The `database finalize` CLI command now supports the `--ram` option, which + controls memory usage for finalization. +- The `database create` CLI command now supports the `--ram` option, which + controls memory usage for database creation. +- The `generate query-help` CLI command now support rendering query help in SARIF format. + ## Release 2.5.7 (2021-07-02) - The bundled extractors are updated to match the versions currently From 484600c2e5fd6226ab7b565fdb8efe6b3558f707 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 9 Aug 2021 22:22:27 +0200 Subject: [PATCH 039/196] changelog for 2.5.9 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 10da2db..e35f3d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,14 @@ you know what to do). --> +## Release 2.5.9 (2021-08-09) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + ## Release 2.5.8 (2021-07-26) - The bundled extractors are updated to match the versions currently From 3f639431d62928b043b19ebfeb7cfe1e1d54814b Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Wed, 25 Aug 2021 00:36:45 +0200 Subject: [PATCH 040/196] changelog for 2.6.0 --- CHANGELOG.md | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e35f3d5..1e5d7f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,89 @@ you know what to do). --> +## Release 2.6.0 (2021-08-24) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Bugs fixed + +- The `physicalLocation.artifactLocation.uri` fields in SARIF output + are now properly encoded as specified by RFC 3986. + +- The `--include-extension` option to the `codeql database + index-files` command no longer includes directories that are named + with the provided extension. For example, if the option + `--include-extension=.rb` is provided, then a directory named + `foo.rb/` will be excluded from the indexing. + +### New features + +- A new `codeql database unbundle` subcommand performs the reverse of + `codeql database bundle` and extracts a CodeQL database from an + archive. + +- The CLI now understands per-codebase configuration files in [the + format already supported by the CodeQL Action][3]. The + configuration file must be given in a `--codescanning-config` option + to `codeql database create` or `codeql database init`. For some + languages, this configuration can contain pathname filters that + control which parts of the codebase is analysed; the configuration + file is the only way this functionality is exposed. The + configuration file can also control which queries are run, including + custom queries from repositories that must first be downloaded. To + actually use those queries, run `codeql database analyze` without + any query-selection arguments. + + [3]: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#example-configuration-files + +- The CLI now supports the "sandwiched tracing" feature that has + previously only been offered through the separate CodeQL Runner. + This feature is intended for use with CI systems that cannot be + configured to wrap build actions with `codeql database + trace-command`. Instead the CI system must be able to set custom + environment variables for each build action; the required + environment variables are output by `codeql database init` when + given a `--begin-tracing` argument. + + On Windows, `codeql database init --begin-tracing` will also inject + build-tracing code into the calling process or an ancestor; there + are additional options to control this. + +- This version contains _beta_ support for a new packaging and + publishing system for third-party QL queries and libraries. It + comprises the following new commands: + + - `codeql pack init`: Creates an empty CodeQL pack from a template. + + - `codeql pack add`: Adds a dependency to a CodeQL pack. + + - `codeql pack install`: Installs all pack dependencies specified in + the `qlpack.yml` file. + + - `codeql pack download`: Downloads one or more pack dependencies + into the global package cache. + + - `codeql pack publish`: Publishes a package to the GitHub Container + Registry. + + - (Plumbing) `codeql pack bundle`: Builds a `.zip` file for a CodeQL + query or library pack from sources. Used by `codeql pack publish`. + + - (Plumbing) `codeql pack create`: Creates a compiled CodeQL query + or library pack from sources. Used by `codeql pack bundle`. + + - (Plumbing) `codeql pack packlist`: Lists all files in a local + CodeQL pack that will be included in the pack's bundle. Used by + `codeql pack create`. + + - (Plumbing) `codeql pack resolve-dependencies`: Resolves all + transitive dependencies of a local CodeQL pack. Used by `codeql + pack install`. + ## Release 2.5.9 (2021-08-09) - The bundled extractors are updated to match the versions currently From f50efa34142a970cbe8d5714d5f2bb52881ce649 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Tue, 7 Sep 2021 16:57:53 -0700 Subject: [PATCH 041/196] changelog for 2.6.1 --- CHANGELOG.md | 103 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e5d7f7..07330a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,109 @@ you know what to do). --> +## Release 2.6.1 (2021-09-07) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Potentially breaking changes + +- The `codeql resolve qlref` command will now throw an error when the + target is ambiguous. + + The qlref resolution rules are now as follows: + + 1. If the target of a qlref is in the same qlpack, then that target is + always returned. + + 2. If multiple targets of the qlref are found in dependent packs, this + is an error. + + Previously, the command would have arbitrarily chosen one of the targets + and ignored any ambiguities. + +- The `qlpack` directive in query suites has its semantics changed. + Previously, this directive would return all queries in the qlpack. Now, + the directive returns only those queries matched by the `defaultSuite` + directive in the query pack. Here is an example: + + Consider a `qlpack.yml` like the following: + + ```yml + name: codeql/my-qlpack + version: 0.0.1 + defaultSuite: + queries: standard + ``` + + And the directory structure is the following: + + ```text + qlpack.yml + standard/ + a.ql + experimental/ + b.ql + ``` + + A query suite `suite.qls` like this: + + ```yml + - qlpack: codeql/my-qlpack + ``` + + Previously, would return all the queries in all subdirectories (i.e, + `standard/a.ql` and `experimental/b.ql`). Now, it only returns + `standard/a.ql`, since that is the only query matched by its default + suite. + + If you want to have the same behavior as before, you must update your + query suites to use the `queries` directive with a `from` attribute, + like this: + + ```yml + - queries: . + from: codeql/my-qlpack + ``` + +### New features + +- Commands that evaluate CodeQL queries now support an additional option + `--evaluator-log=path/to/log.json` that will result in the evaluator + producing a structured log (in JSON format) of events that occurred + during evaluation in order to aid debugging of query performance. The + format of these logs will be subject to change with no notice as we + make modifications to the evaluator. + + There is also a new CLI command `codeql generate log-summary` that will + produce a summary of the predicates that were evaluated from these event + logs. We will aim to keep this summary format more stable, although it + is also subject to change. Unless you have a good reason to use the + event logs directly, it is strongly recommended you use this command to + produce summary logs and use these instead. + + For further information on these new logs and additional options to + configure their format and verbosity, please refer to the CLI + documentation. + +### New language features + +- QL classes can now be non-extending subtypes via the `instanceof` + keyword, allowing for a form of private subtyping that is not visible + externally. Methods of the supertype are accessible from within a + non-extending subtype class through extended semantics of the `super` + keyword. + + ``` + class Foo instanceof int { + Foo() { this in [1 .. 10] } + string toString() { result = "foo" + super.toString() } + } + ``` + ## Release 2.6.0 (2021-08-24) - The bundled extractors are updated to match the versions currently From d7b08e6cfc3d29105c1b4d73c8ef5502f4ee2909 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano <6748066+edoardopirovano@users.noreply.github.com> Date: Tue, 21 Sep 2021 17:51:14 +0100 Subject: [PATCH 042/196] Update changelog for 2.6.2 release --- CHANGELOG.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 07330a7..254cab1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,57 @@ you know what to do). --> +## Release 2.6.2 (2021-09-21) + +- CodeQL CLI 2.6.2 includes the same functionality as **the CodeQL runner**, + which is being deprecated. For more information, see [CodeQL runner + deprecation][5]. + + [5]: https://github.blog/changelog/2021-09-21-codeql-runner-deprecation/ + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.27) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.27 instance, you need to create them with release + 2.4.6. + +### Bugs fixed + +- A bug where `codeql generate log-summary` would sometimes crash with + a `JsonMappingException` has been fixed. + +### New features + +- The CodeQL CLI now counts the lines of code found under `--source-root` + when `codeql database init` or `codeql database create` is called. This + information can be viewed later by either the new + `codeql database print-baseline` command or the new `--print-baseline-loc` + argument to `codeql database interpret-results`. +- `qlpack.yml` files now support an additional field `include` in which + glob patterns of additional files that should be included (or excluded) + when creating a given CodeQL pack can be specified. +- QL packs created by the experimental `codeql pack create` command will + now include some information about the build in a new `buildMetadata` + field of their `qlpack.yml` file. +- `codeql database create` now supports the same flags as `codeql database init` + for automatically recognizing the languages present in checkouts of GitHub + repositories: + - `--github-url` accepts the URL of a custom GitHub instance (previously + only `github.com` was supported). + - `--github-auth-stdin` allows a personal access token to be provided + through standard input (previously only the `GITHUB_TOKEN` environment + variable was supported). + +### Notable documentation changes + +- Documentation has been added detailing how to use the "indirect build + tracing" feature, which is enabled by using the `--begin-tracing` flag + provided by `codeql database init`. The new documentation can be found + [here][4]. This feature was temporarily described as "sandwiched tracing" + in the 2.6.0 release notes. + + [4]: https://aka.ms/codeql-docs/indirect-tracing + ## Release 2.6.1 (2021-09-07) - The bundled extractors are updated to match the versions currently From 0970517e0285c46856868c75ec58bd12a12e6435 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Wed, 6 Oct 2021 18:19:57 +0100 Subject: [PATCH 043/196] Update changelog for 2.6.3 --- CHANGELOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 254cab1..33b5536 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,49 @@ you know what to do). --> +## Release 2.6.3 (2021-10-06) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Potentially breaking changes +- The option `--compiler-spec` accepted by some subcommands of `codeql database` is deprecated. + It will be removed in a later version (earliest 2.7.0). + If you need this option, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering. +- By default, databases created using the CodeQL CLI will now have their underlying datasets finalized, meaning that no + further data can be subsequently imported into them. This change should not affect most users. +- The `codeql resolve qlref` command will now throw an error when the target is ambiguous. + The qlref resolution rules are now as follows: + 1. If the target of a qlref is in the same qlpack, then that target is always returned. + 2. If multiple targets of the qlref are found in dependent packs, this is an error. + + Previously, the command would have arbitrarily chosen one of the targets and ignored any + ambiguities. + + +### Bugs fixed +- Linux/MacOS: When tracing a build that involves an `execvp`/`execvpe` (Linux-only)/`posix_spawnp` syscall + where `PATH` was not set in the environment, CodeQL sometimes would break the build. + Now, CodeQL uses the correct, platform-specific fallback for `PATH` instead. +- Linux/MacOS: When tracing a build that involves an `execvpe` (Linux-only)/`posix_spawnp` syscall, + the `PATH` lookup of the executable wrongly took place in the environment provided via `envp`, + instead of the environment of the process calling `execvpe`/`posix_spawnp`. + Now, the correct environment is used for the `PATH` lookup. +- A bug where query compilation would sometimes fail with a `StackOverflowError` when compiling a query that + uses `instanceof` has now been fixed. + + +### New features + +- The `codeql query compile` command now accepts a `--keep-going` or `-k` option, which indicates that the compiler should continue compiling queries even if one of the queries has a compile error in it. +- CLI commands now run default queries if none are specified. If no queries are specified, the `codeql database analyze`, `codeql database run-queries`, + and `codeql database interpret-results` commands will now run the default suite for the language being analyzed. +- `codeql pack publish` now copies the published package to the local package cache. In addition to publishing to a remote repository, the `codeql pack publish` command will also copy the published package to the local package cache. + + ## Release 2.6.2 (2021-09-21) - CodeQL CLI 2.6.2 includes the same functionality as **the CodeQL runner**, From 47bcdd4ed3bcc5a2e0ad8583941ac6d4e8a66401 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Wed, 27 Oct 2021 14:07:15 -0700 Subject: [PATCH 044/196] changelog for 2.7.0 --- CHANGELOG.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33b5536..79286aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,24 @@ you know what to do). --> +## Release 2.7.0 (2021-10-27) + +- The extractor for Ruby is now included. CodeQL analysis for Ruby is + currently in beta. During the beta, analysis of Ruby will not be as + comprehensive as CodeQL analysis of other languages. The source code + of the extractor and the queries can be found in the + [`github/codeql`](https://github.com/codeql) repository. +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Bugs fixed + +- Fixed a bug where indirect tracing would sometimes not manage to observe + build processes if certain environment variables were unset during the build. + ## Release 2.6.3 (2021-10-06) - The bundled extractors are updated to match the versions currently From e6e232af6143f54c78fa893fdde35efd285a7d68 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Wed, 27 Oct 2021 15:13:20 -0700 Subject: [PATCH 045/196] Fix broken link to github/codeql --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 79286aa..aeb190a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,7 +23,7 @@ currently in beta. During the beta, analysis of Ruby will not be as comprehensive as CodeQL analysis of other languages. The source code of the extractor and the queries can be found in the - [`github/codeql`](https://github.com/codeql) repository. + [`github/codeql`](https://github.com/github/codeql) repository. - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM From 962cb01301a7e4590488f43dd1e3e394c9210146 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 28 Oct 2021 23:13:04 +0200 Subject: [PATCH 046/196] backpatch changelog: release 2.5.9 went with LGTM-e 1.28 --- CHANGELOG.md | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aeb190a..f2cc8f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,10 +87,10 @@ [5]: https://github.blog/changelog/2021-09-21-codeql-runner-deprecation/ - The bundled extractors are updated to match the versions currently - used on LGTM.com. These are newer than the last release (1.27) of + used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM - Enterprise 1.27 instance, you need to create them with release - 2.4.6. + Enterprise 1.28 instance, you need to create them with release + 2.5.9. ### Bugs fixed @@ -132,10 +132,10 @@ ## Release 2.6.1 (2021-09-07) - The bundled extractors are updated to match the versions currently - used on LGTM.com. These are newer than the last release (1.27) of + used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM - Enterprise 1.27 instance, you need to create them with release - 2.4.6. + Enterprise 1.28 instance, you need to create them with release + 2.5.9. ### Potentially breaking changes @@ -235,10 +235,10 @@ ## Release 2.6.0 (2021-08-24) - The bundled extractors are updated to match the versions currently - used on LGTM.com. These are newer than the last release (1.27) of + used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM - Enterprise 1.27 instance, you need to create them with release - 2.4.6. + Enterprise 1.28 instance, you need to create them with release + 2.5.9. ### Bugs fixed @@ -317,11 +317,10 @@ ## Release 2.5.9 (2021-08-09) -- The bundled extractors are updated to match the versions currently - used on LGTM.com. These are newer than the last release (1.27) of - LGTM Enterprise. If you plan to upload databases to an LGTM - Enterprise 1.27 instance, you need to create them with release - 2.4.6. +This release corresponds to release 1.28.x of LGTM Enterprise, and +should be used when creating databases that will be uploaded to it. +Future CLI releases (numbered 2.6.x) may produce databases that are not +backwards compatible with this version of LGTM Enterprise. ## Release 2.5.8 (2021-07-26) From 84a221dcf6776f22aaa9191017cb180cb1ea527c Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 15 Nov 2021 20:13:25 +0100 Subject: [PATCH 047/196] changelog for 2.7.1 --- CHANGELOG.md | 373 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 288 insertions(+), 85 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f2cc8f4..4db30ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,167 @@ you know what to do). --> +## Release 2.7.1 (2021-11-15) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Potentially breaking changes + +- Previously, `codeql test run` would fall back to looking for an + accompanying `queries.xml` file if it found a `qlpack.yml` that did + not declare an extractor to use when extracting a test database. + This has been removed because the internal use case that + neccessitated the fallback are now removed. If you suddenly + encounter errors that complain of missing extractor declarations, + check whether you had a `queries.xml` you were inadvertently relying + on. + +- When queries are specified by naming a directory to scan for `*.ql` + files, subdirectories named `.codeql` will now be ignored. The new + QL packaging support uses subdirectories with this name of various + scratch and caching purposes, so they may contain `*.ql` files that + are not intended to be directly user-visible. + +- When copying dependencies for CodeQL packages into a query pack + bundle, `*.ql` files in these dependencies will now be included + inside of the query pack's `.codeql` directory. + +- The tables printed by `codeql database analyze` to summarize the + results of diagnostic and metric queries that were part of the + analysis have a new format and contains less (but hopefully more + pertinent) information. We recommend against attempting to parse + this human-readable output programmatically. Instead, use the + `runs[].tool.driver.invocations[].toolExecutionNotifications` + property in the SARIF output. + +- The experimental plumbing command `codeql pack packlist` has a new + format for its JSON results. Previously, the results were a list of + paths. Now, the results are an object with a single property `paths` + that contains the list of paths. + +### Deprecations + +- The output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee + Specification Draft 1) have been deprecated. They will be removed + in a later version (earliest 2.8.0). If you need this + functionality, please file a public issue against + https://github.com/github/codeql-cli-binaries, or open a private + ticket with GitHub Support and request an escalation to engineering. + +- The `qlpack:` instruction in query suite definitions has been + deprecated due to uncertainty about whether it is intended to + include _all_ the `*.ql` files in the named pack, or only the pack's + "default query suite". The behavior of the instruction is + determined by whether the named pack declares any default query + suite, but this means that a pack _starting_ to declare such a suite + may break the behavior of existing query suites that reference the + pack from outside. + + We recommend replacing `qlpack:` by one of + ```yaml + - queries: '.' # import all *.ql files + from: some/pack-name + version: 1.2.3 # optional + ``` + or + ```yaml + - import: path/to/actual/suite.ql # just that suite + from: some/pack-name + version: 1.2.3 # optional + ``` + + A warning will now be printed when a `qlpack:` instruction resolves + to a default suite, because that is the case where the effect may + not be what the query suite author intended. + +### Bugs fixed + +- Fixed a bug where the `paths` and `paths-ignore` properties of a + Code Scanning config file specified using `--codescanning-config` + were being interpreted the wrong way around. + +- Fixed a bug where queries specified using the + `--codescanning-config` option could not be run after an explicit + call to `codeql database finalize`. + +- Fixed a bug where `-J` options would erroneously be recognized even + after `--` on the command line. + +- When running `codeql database analyze` and `codeql database + interpret-results` without the `--sarif-group-rules-by-pack` flag, + the SARIF output did not include baseline lines-of-code counts. This + is now fixed. + +- Fixed a bug where expansion of query suites would sometimes fail if + a query suite in a compiled query pack referenced that pack itself + explicitly. + +### New language features + +- [Set literal expressions][9] can now optionally contain a trailing + comma after the last element. + + [9]: https://codeql.github.com/docs/ql-language-reference/expressions/#set-literal-expressions + +### New features + +- Beta support for database creation on Apple Silicon has been added. + It depends on the following requirements: + + - [Rosetta 2][8] needs to be installed + + [8]: https://developer.apple.com/documentation/apple-silicon/about-the-rosetta-translation-environment + + - Developer tools need to be installed. CodeQL requires the `lipo`, + `codesign`, and `install_name_tool` tools to be present. + + - Build systems invoking `csh` may experience [intermittent + crashes][7]. + + [7]: https://openradar.appspot.com/radar?id=4936797431791616 + +- `codeql database analyze` can now include query-specific help texts + for alerts in the SARIF output (for SARIF v2.1.0 or later). The help + text must be located in an `.md` file next to (and with the same + basename as) the `.ql` file for each query. Since this can + significantly increase SARIF file size, the feature is not enabled + by default; give a `--sarif-add-query-help` option to enable it. + +- The query metadata validator now knows about queries that produce + alert scores, so these queries no longer need to be run with a + `--no-metadata-verification` flag. + +- `codeql database create` and `codeql-finalize` have a new flag + `--skip-empty` that will cause a language with no extracted source + code to be ignored with a warning instead of treated like a fatal + error. This can be useful with `--db-cluster` where not all of the + languages may exist in the source tree. It will not be possible to + run queries against the skipped database. + +- `codeql resolve extractor` and `codeql resolve languages` now + support an extended output format `--format=betterjson` wich + includes information about each extractor's language-specific + options. + +- This release introduces rudimentary support for parallelizing + database creation by importing unfinished databases (or database + clusters) into another unfinished database (or cluster) under + creation. This is implemented by the new flag `--additional-dbs` for + `codeql database finalize`, or the new plumbing command `codeql + database import`. + +- `codeql database create`, `codeql database index-files`, and `codeql + database trace-command` support a [unified syntax for passing + language-specific options][6] to the extractor with the new + `--extractor-option` and `--extractor-options-file` options. + (The extractors do not make use of this yet, though). + + [6]: https://codeql.github.com/docs/codeql-cli/extractor-options + ## Release 2.7.0 (2021-10-27) - The extractor for Ruby is now included. CodeQL analysis for Ruby is @@ -24,6 +185,7 @@ comprehensive as CodeQL analysis of other languages. The source code of the extractor and the queries can be found in the [`github/codeql`](https://github.com/github/codeql) repository. + - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM @@ -32,8 +194,9 @@ ### Bugs fixed -- Fixed a bug where indirect tracing would sometimes not manage to observe - build processes if certain environment variables were unset during the build. +- Fixed a bug where indirect tracing would sometimes not manage to + observe build processes if certain environment variables were unset + during the build. ## Release 2.6.3 (2021-10-06) @@ -44,45 +207,73 @@ 2.5.9. ### Potentially breaking changes -- The option `--compiler-spec` accepted by some subcommands of `codeql database` is deprecated. - It will be removed in a later version (earliest 2.7.0). - If you need this option, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering. -- By default, databases created using the CodeQL CLI will now have their underlying datasets finalized, meaning that no - further data can be subsequently imported into them. This change should not affect most users. -- The `codeql resolve qlref` command will now throw an error when the target is ambiguous. - The qlref resolution rules are now as follows: - 1. If the target of a qlref is in the same qlpack, then that target is always returned. - 2. If multiple targets of the qlref are found in dependent packs, this is an error. + +- The option `--compiler-spec` accepted by some subcommands of `codeql + database` is deprecated. It will be removed in a later version + (earliest 2.7.0). If you need this option, please file a public + issue in https://github.com/github/codeql-cli-binaries, or open a + private ticket with GitHub support and request an escalation to + engineering. + +- By default, databases created using the CodeQL CLI will now have + their underlying datasets finalized, meaning that no further data + can be subsequently imported into them. This change should not + affect most users. + +- The `codeql resolve qlref` command will now throw an error when the + target is ambiguous. The qlref resolution rules are now as follows: + + 1. If the target of a qlref is in the same qlpack, then that target + is always returned. + + 2. If multiple targets of the qlref are found in dependent packs, + this is an error. Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities. - ### Bugs fixed -- Linux/MacOS: When tracing a build that involves an `execvp`/`execvpe` (Linux-only)/`posix_spawnp` syscall - where `PATH` was not set in the environment, CodeQL sometimes would break the build. - Now, CodeQL uses the correct, platform-specific fallback for `PATH` instead. -- Linux/MacOS: When tracing a build that involves an `execvpe` (Linux-only)/`posix_spawnp` syscall, - the `PATH` lookup of the executable wrongly took place in the environment provided via `envp`, - instead of the environment of the process calling `execvpe`/`posix_spawnp`. - Now, the correct environment is used for the `PATH` lookup. -- A bug where query compilation would sometimes fail with a `StackOverflowError` when compiling a query that - uses `instanceof` has now been fixed. +- Linux/MacOS: When tracing a build that involves an + `execvp`/`execvpe` (Linux-only)/`posix_spawnp` syscall where `PATH` + was not set in the environment, CodeQL sometimes would break the + build. Now, CodeQL uses the correct, platform-specific fallback for + `PATH` instead. + +- Linux/MacOS: When tracing a build that involves an `execvpe` + (Linux-only)/`posix_spawnp` syscall, the `PATH` lookup of the + executable wrongly took place in the environment provided via + `envp`, instead of the environment of the process calling + `execvpe`/`posix_spawnp`. Now, the correct environment is used for + the `PATH` lookup. + +- A bug where query compilation would sometimes fail with a + `StackOverflowError` when compiling a query that uses `instanceof` + has now been fixed. ### New features -- The `codeql query compile` command now accepts a `--keep-going` or `-k` option, which indicates that the compiler should continue compiling queries even if one of the queries has a compile error in it. -- CLI commands now run default queries if none are specified. If no queries are specified, the `codeql database analyze`, `codeql database run-queries`, - and `codeql database interpret-results` commands will now run the default suite for the language being analyzed. -- `codeql pack publish` now copies the published package to the local package cache. In addition to publishing to a remote repository, the `codeql pack publish` command will also copy the published package to the local package cache. +- The `codeql query compile` command now accepts a `--keep-going` or + `-k` option, which indicates that the compiler should continue + compiling queries even if one of the queries has a compile error in + it. + +- CLI commands now run default queries if none are specified. If no + queries are specified, the `codeql database analyze`, `codeql + database run-queries`, and `codeql database interpret-results` + commands will now run the default suite for the language being + analyzed. +- `codeql pack publish` now copies the published package to the local + package cache. In addition to publishing to a remote repository, the + `codeql pack publish` command will also copy the published package + to the local package cache. ## Release 2.6.2 (2021-09-21) -- CodeQL CLI 2.6.2 includes the same functionality as **the CodeQL runner**, - which is being deprecated. For more information, see [CodeQL runner - deprecation][5]. +- CodeQL CLI 2.6.2 includes the same functionality as **the CodeQL + runner**, which is being deprecated. For more information, see + [CodeQL runner deprecation][5]. [5]: https://github.blog/changelog/2021-09-21-codeql-runner-deprecation/ @@ -99,33 +290,39 @@ ### New features -- The CodeQL CLI now counts the lines of code found under `--source-root` - when `codeql database init` or `codeql database create` is called. This - information can be viewed later by either the new - `codeql database print-baseline` command or the new `--print-baseline-loc` - argument to `codeql database interpret-results`. -- `qlpack.yml` files now support an additional field `include` in which - glob patterns of additional files that should be included (or excluded) - when creating a given CodeQL pack can be specified. -- QL packs created by the experimental `codeql pack create` command will - now include some information about the build in a new `buildMetadata` - field of their `qlpack.yml` file. -- `codeql database create` now supports the same flags as `codeql database init` - for automatically recognizing the languages present in checkouts of GitHub - repositories: - - `--github-url` accepts the URL of a custom GitHub instance (previously - only `github.com` was supported). - - `--github-auth-stdin` allows a personal access token to be provided - through standard input (previously only the `GITHUB_TOKEN` environment - variable was supported). +- The CodeQL CLI now counts the lines of code found under + `--source-root` when `codeql database init` or `codeql database + create` is called. This information can be viewed later by either + the new `codeql database print-baseline` command or the new + `--print-baseline-loc` argument to `codeql database + interpret-results`. + +- `qlpack.yml` files now support an additional field `include` in + which glob patterns of additional files that should be included (or + excluded) when creating a given CodeQL pack can be specified. + +- QL packs created by the experimental `codeql pack create` command + will now include some information about the build in a new + `buildMetadata` field of their `qlpack.yml` file. + +- `codeql database create` now supports the same flags as `codeql + database init` for automatically recognizing the languages present + in checkouts of GitHub repositories: + + - `--github-url` accepts the URL of a custom GitHub instance + (previously only `github.com` was supported). + + - `--github-auth-stdin` allows a personal access token to be + provided through standard input (previously only the + `GITHUB_TOKEN` environment variable was supported). ### Notable documentation changes -- Documentation has been added detailing how to use the "indirect build - tracing" feature, which is enabled by using the `--begin-tracing` flag - provided by `codeql database init`. The new documentation can be found - [here][4]. This feature was temporarily described as "sandwiched tracing" - in the 2.6.0 release notes. +- Documentation has been added detailing how to use the "indirect + build tracing" feature, which is enabled by using the + `--begin-tracing` flag provided by `codeql database init`. The new + documentation can be found [here][4]. This feature was temporarily + described as "sandwiched tracing" in the 2.6.0 release notes. [4]: https://aka.ms/codeql-docs/indirect-tracing @@ -154,9 +351,9 @@ and ignored any ambiguities. - The `qlpack` directive in query suites has its semantics changed. - Previously, this directive would return all queries in the qlpack. Now, - the directive returns only those queries matched by the `defaultSuite` - directive in the query pack. Here is an example: + Previously, this directive would return all queries in the + qlpack. Now, the directive returns only those queries matched by the + `defaultSuite` directive in the query pack. Here is an example: Consider a `qlpack.yml` like the following: @@ -199,19 +396,20 @@ ### New features -- Commands that evaluate CodeQL queries now support an additional option - `--evaluator-log=path/to/log.json` that will result in the evaluator - producing a structured log (in JSON format) of events that occurred - during evaluation in order to aid debugging of query performance. The - format of these logs will be subject to change with no notice as we - make modifications to the evaluator. - - There is also a new CLI command `codeql generate log-summary` that will - produce a summary of the predicates that were evaluated from these event - logs. We will aim to keep this summary format more stable, although it - is also subject to change. Unless you have a good reason to use the - event logs directly, it is strongly recommended you use this command to - produce summary logs and use these instead. +- Commands that evaluate CodeQL queries now support an additional + option `--evaluator-log=path/to/log.json` that will result in the + evaluator producing a structured log (in JSON format) of events that + occurred during evaluation in order to aid debugging of query + performance. The format of these logs will be subject to change with + no notice as we make modifications to the evaluator. + + There is also a new CLI command `codeql generate log-summary` that + will produce a summary of the predicates that were evaluated from + these event logs. We will aim to keep this summary format more + stable, although it is also subject to change. Unless you have a + good reason to use the event logs directly, it is strongly + recommended you use this command to produce summary logs and use + these instead. For further information on these new logs and additional options to configure their format and verbosity, please refer to the CLI @@ -220,10 +418,10 @@ ### New language features - QL classes can now be non-extending subtypes via the `instanceof` - keyword, allowing for a form of private subtyping that is not visible - externally. Methods of the supertype are accessible from within a - non-extending subtype class through extended semantics of the `super` - keyword. + keyword, allowing for a form of private subtyping that is not + visible externally. Methods of the supertype are accessible from + within a non-extending subtype class through extended semantics of + the `super` keyword. ``` class Foo instanceof int { @@ -331,19 +529,24 @@ backwards compatible with this version of LGTM Enterprise. 2.4.6. ### Potentially breaking changes -- The QL compiler now verifies that `@security-severity` query metadata is numeric. You can disable - this verification by passing the `--no-metadata-verification` flag. + +- The QL compiler now verifies that `@security-severity` query + metadata is numeric. You can disable this verification by passing + the `--no-metadata-verification` flag. ### New features -- The `database index-files` and `database trace-command` CLI commands now - support `--threads` and `--ram` options, which are passed to extractors as - suggestions. -- The `database finalize` CLI command now supports the `--ram` option, which - controls memory usage for finalization. -- The `database create` CLI command now supports the `--ram` option, which - controls memory usage for database creation. -- The `generate query-help` CLI command now support rendering query help in SARIF format. +- The `database index-files` and `database trace-command` CLI commands + now support `--threads` and `--ram` options, which are passed to + extractors as suggestions. + +- The `database finalize` CLI command now supports the `--ram` option, + which controls memory usage for finalization. + +- The `database create` CLI command now supports the `--ram` option, + which controls memory usage for database creation. - The `generate + query-help` CLI command now support rendering query help in SARIF + format. ## Release 2.5.7 (2021-07-02) @@ -373,9 +576,9 @@ backwards compatible with this version of LGTM Enterprise. ### New language features - The QL language now has a new method `toUnicode` on the `int` - type. This method converts Unicode codepoint to a one-character string. - For example, `65.toUnicode() = "A"`, `128512.toUnicode()` results in - a smiley, and `any(int i | i.toUnicode() = "A") = 65`. + type. This method converts Unicode codepoint to a one-character + string. For example, `65.toUnicode() = "A"`, `128512.toUnicode()` + results in a smiley, and `any(int i | i.toUnicode() = "A") = 65`. ## Release 2.5.6 (2021-06-22) From 2953d2a66ca9fdbfe7e447f650206f9549e09307 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Mon, 22 Nov 2021 13:41:58 +0000 Subject: [PATCH 048/196] changelog for 2.7.2 --- CHANGELOG.md | 47 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4db30ae..131bb65 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,49 @@ you know what to do). --> +## Release 2.7.2 (2021-11-22) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Potentially breaking changes + +- The Java extractor now defaults to extracting all XML documents under + 10MB in size, a change from the previous default of only extracting + documents with particular well-known names (e.g. `pom.xml`). However, + if the source tree contains more than 50MB of XML in total, it prints + a warning and falls back to the old default behaviour. + Set the environment variable `LGTM_INDEX_XML_MODE` to `byname` to get + the old default behaviour, or `all` to extract all documents under + 10MB regardless of total size. + +- The experimental command-line option `--native-library-path` that was + introduced to support internal experiments has been removed. + +- The beta `codeql pack publish` command will now prevent accidental + publishing of packages with pre-release version qualifiers. Prerelease + versions are those that include a `-` after the major, minor, and patch + versions such as `1.2.3-dev`. To avoid this change, use the + `--allow-prerelease` option. + +### Bugs fixed + +- Fixed an issue when using the `--evaluator-log` option where a + `NullPointerException` could sometimes occur non-deterministically. + +- Fixed bugs observed when using indirect build tracing using a CodeQL + distribution unpacked to a path containing spaces or on Arch Linux. + +### New features + +- CodeQL databases now contain metadata about how and when they were + created. This can be found in the `creationMetadata` field of the + `codeql-database.yml` file within the CodeQL database directory. More + information may be added to this field in future releases. + ## Release 2.7.1 (2021-11-15) - The bundled extractors are updated to match the versions currently @@ -228,7 +271,7 @@ 2. If multiple targets of the qlref are found in dependent packs, this is an error. - + Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities. @@ -535,7 +578,7 @@ backwards compatible with this version of LGTM Enterprise. the `--no-metadata-verification` flag. ### New features - + - The `database index-files` and `database trace-command` CLI commands now support `--threads` and `--ram` options, which are passed to extractors as suggestions. From 8a8f2f036a16f1201292e91d5973385519dffd13 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Wed, 1 Dec 2021 09:28:07 -0800 Subject: [PATCH 049/196] Retroactively add a note about the qlpack structure change --- CHANGELOG.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 131bb65..3f6c104 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,6 +101,30 @@ format for its JSON results. Previously, the results were a list of paths. Now, the results are an object with a single property `paths` that contains the list of paths. + +- The internal `qlpacks` directory of the CodeQL bundle available on the + [CodeQL Action releases page](https://github.com/github/codeql-action/releases/) + has a new structure. This directory is internal to the CLI and can change without notice in future releases. + + The currently-shipped `qlpacks` directory mirrors the structure of [CodeQL package](https://github.blog/changelog/2021-07-28-introducing-the-codeql-package-manager-public-beta/) caches and looks like this: + + ```text + qlpacks + - codeql + - {lang}-all + - {version} + - qlpack contents + - {lang}-examples + - {version} + - qlpack contents + - {lang}-queries + - {version} + - qlpack contents + - {lang}-upgrades + - {version} + - qlpack contents + - ... and so on for all languages + ``` ### Deprecations From e7a42a08f360970af823dd7ea6189ca5ea234ffe Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 7 Dec 2021 11:06:44 -0800 Subject: [PATCH 050/196] Update CHANGELOG.md for 2.7.3 --- CHANGELOG.md | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 51 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3f6c104..76c79a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,54 @@ you know what to do). --> +## Release 2.7.3 (2021-12-06) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Potentially breaking changes + +- The experimental command-line option `--ml-model-path` that was + introduced to support internal experiments has been removed. + +### Bugs fixed + +- Editing support (content assist, code navigation, etc.) in files + under the `.github` directory will now work properly. This is + because files under the `.github` directory will now be indexed and + processed by the CodeQL language server. Other hidden directories + that start with `.` will remain un-indexed. This affects the + vscode-codeql extension and any other IDE extension that uses + the CodeQL language server. + +- Fixed authentication with GitHub package registries via the + `GITHUB_TOKEN` environment variable and the `--github-auth-stdin` + flag when downloading and publishing packs. + +- Fixed an incompatibility with glibc version 2.34 on Linux, where + build tracing failed with an error message. + +- Fixed a bug where `codeql generate log-summary` could sometimes fail + with a `JsonMappingException`. + +### New features + +- The CodeQL CLI for Mac OS now ships with a native Java virtual machine for M1 Macs, + and this will be used by default where applicable to run the CodeQL + engine, thus improving performance. + [Rosetta 2](https://support.apple.com/en-us/HT211861) is still + required as not all components of the CodeQL CLI are natively compiled. + +- Commands that execute queries will now exit with status code 34 if + certain errors that prevent the evaluation of one or more + individual queries are detected. Previously some of these errors + would crash the evaluator and exit with status code 100. + + (This is currently used for "external predicate not found" errors). + ## Release 2.7.2 (2021-11-22) - The bundled extractors are updated to match the versions currently @@ -101,10 +149,11 @@ format for its JSON results. Previously, the results were a list of paths. Now, the results are an object with a single property `paths` that contains the list of paths. - + - The internal `qlpacks` directory of the CodeQL bundle available on the [CodeQL Action releases page](https://github.com/github/codeql-action/releases/) - has a new structure. This directory is internal to the CLI and can change without notice in future releases. + has a new structure. This directory is internal to the CLI and can change without + notice in future releases. The currently-shipped `qlpacks` directory mirrors the structure of [CodeQL package](https://github.blog/changelog/2021-07-28-introducing-the-codeql-package-manager-public-beta/) caches and looks like this: From 4d7fd805687040155c0510398d98a889b91ed4cb Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 17 Jan 2022 19:51:05 +0100 Subject: [PATCH 051/196] Changelog for 2.7.5 --- CHANGELOG.md | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 76c79a1..3b9a8c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,59 @@ you know what to do). --> +## Release 2.7.5 (2022-01-17) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Deprecation + +- The CodeQL Action versions up to and including version 1.0.22 are + now deprecated for use with CodeQL CLI 2.7.5 and later. The CLI + will emit a warning if it detects that it is being used by a + deprecated version of the codeql-action. This warning will become a + fatal error with version 2.8.0 of the CLI. + +### New feature + +- The `codeql github upload-results` command will now print the API + response body in JSON format if a `--format=json` flag is + given. Otherwise the command will print the URL of the SARIF + upload. This URL can be used to get status information for the + upload. + + See also: https://docs.github.com/en/rest/reference/code-scanning + +### Documentation fixes + +- The documentation for the `--trace-process-level` flag of `codeql + database init` (which is used with indirect build tracing on + Windows) was erroneous. + + The help text previously claimed that `--trace-process-level=1` + would inject CodeQL's build tracer into the calling process. This is + actually what `--trace-process-level=0` achieves. The help text has + now been corrected to match the actual (unchanged) behavior. + + Also, some log messages incorrectly stated which process CodeQL was + injected into. These have also been corrected. + +### Other changes + +- For commands that run queries, the `--timeout` option now controls + the maximal time it may take to evaluate a "layer" of a query rather + than a "stage". There are usually many "layers" in each "stage", + but it is usually a single one of the layers in a stage that uses + most of the time, so there is no need to reduce existing timeout + values as a result of this change. + +## Release 2.7.4 + +This release was skipped. + ## Release 2.7.3 (2021-12-06) - The bundled extractors are updated to match the versions currently From 5afcd8509518e687e1a25e58eca9c2eef60957a2 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Mon, 24 Jan 2022 11:26:46 +0000 Subject: [PATCH 052/196] Update CHANGELOG for 2.7.6 release --- CHANGELOG.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b9a8c1..6e6419d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,27 @@ you know what to do). --> +## Release 2.7.6 (2022-01-24) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.28) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.28 instance, you need to create them with release + 2.5.9. + +### Bug fixed + +- A bug where creation of a CodeQL database could sometimes fail with + a `NegativeArraySizeException` has now been fixed. + +### New feature + +- The CLI and evaluator contain a number of new features in support of + internal machine learning experiments. This includes an experimental + `resolve ml-models` subcommand and new `mlModels` metadata in pack + definition files. As these new features are not yet ready for general + use, they should be ignored by external CodeQL users. + ## Release 2.7.5 (2022-01-17) - The bundled extractors are updated to match the versions currently From 002c419626cc5e5d9cce705d78b93f26d77835e5 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Fri, 4 Feb 2022 13:09:10 -0800 Subject: [PATCH 053/196] Update CHANGELOG for 2.8.0 release --- CHANGELOG.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e6419d..b614ee4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,43 @@ you know what to do). --> +## Release 2.8.0 (2022-02-04) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.29) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.29 instance, you need to create them with release + 2.6.3. + +### Breaking change + +- The CodeQL Action versions up to and including version 1.0.22 are + not compatible with the CodeQL CLI 2.8.0 and later. The CLI + will emit an error if it detects that it is being used by an + incompatible version of the codeql-action. + +### New features + +- A new extractor option has been added to the Java extractor. The + flag `--extractor-option exclude=''` allows specifying a glob + that describes which paths need to be excluded from extraction but + still need to be compiled. This is useful when some files are necessary + for a successful build but are uninteresting for analysis. + + See also: https://codeql.github.com/docs/codeql-cli/extractor-options/ + +- Summary metrics can now associate messages with their results, for + instance to report the name and number of uses of a particular API + endpoint within a repository. To associate messages with summary + metrics, define a query with `@kind metric` and `@tags summary` metadata + and use either the `location, message, value` or the `message, value` + results pattern. + +### Bug fixed + +- Fixed a bug where `codeql resolve upgrades` ignores the + `--target-dbscheme` option. + ## Release 2.7.6 (2022-01-24) - The bundled extractors are updated to match the versions currently From 76ac347d67fa0b65bed3338514e8e86a8012c10c Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 15 Feb 2022 15:11:31 +0000 Subject: [PATCH 054/196] Update changelog for 2.8.1 --- CHANGELOG.md | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b614ee4..8f25f89 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,67 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.8.1 (2022-02-15) +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.29) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.29 instance, you need to create them with release + 2.6.3. + +### New Features + +- Commands that find or run queries now allow you to refer to queries within a named CodeQL + pack. For example: + + ```sh + # Analyze a database using all queries in the experimental/Security folder within the codeql/cpp-queries + # CodeQL query pack. + codeql database analyze --format=sarif-latest --output=results \ + codeql/cpp-queries:experimental/Security + + # Analyse using only the RedundantNullCheckParam.ql query in the codeql/cpp-queries CodeQL query pack. + codeql database analyze --format=sarif-latest --output=results \ + 'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql' + + # Analyse using the cpp-security-and-quality.qls query suite in the codeql/cpp-queries CodeQL query pack. + codeql database analyze --format=sarif-latest --output=results \ + 'codeql/cpp-queries:codeql-suites/cpp-security-and-quality.qls' + + # Analyse using the cpp-security-and-quality.qls query suite from a version of the codeql/cpp-queries pack + # that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen). + # All valid semver ranges are allowed. See https://docs.npmjs.com/cli/v6/using-npm/semver#ranges + codeql database analyze --format=sarif-latest --output=results \ + 'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls' + ``` + + The complete way to specify a set of queries is in the form `scope/name@range:path`, where: + + - `scope/name` is the qualified name of a CodeQL pack. + - `range` is a [semver range](https://docs.npmjs.com/cli/v6/using-npm/semver#ranges). + - `path` is a file system path + + If a `scope/name` is specified, the `range` and `path` are optional. A missing `range` + implies the latest version of the specified pack. A missing `path` implies the default + query suite of the specified pack. + + The `path` can be one of a `*.ql` query file, a directory containing one or more queries, or a + `.qls` query suite file. If there is no pack name specified, then a `path` must be provided, and will + be interpreted relative to the current working directory of the current process. + + If a `scope/name` and `path` are specified, then the `path` cannot be absolute. It is considered + relative to the root of the CodeQL pack. + + The relevant commands are: + - `codeql database analyze` + - `codeql database run-queries` + - `codeql execute queries` + - `codeql resolve queries` + +### Bugs fixed + +- Fixed a bug that would sometimes lead to query evaluation on + M1-based Macs to crash with `Did not preallocate enough + memory` error. ## Release 2.8.0 (2022-02-04) From bb23096ce83d873c978e015d7ee142689ffa64d4 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Mon, 28 Feb 2022 19:27:48 +0100 Subject: [PATCH 055/196] Changelog for 2.8.2 --- CHANGELOG.md | 59 +++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 45 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f25f89..7ef2aab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,31 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.8.2 (2022-02-28) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.29) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.29 instance, you need to create them with release + 2.6.3. + +### Breaking change + +- The support for the output formats SARIF v1.0.0 and SARIF v2.0.0 + (Committee Specification Draft 1) that were deprecated in 2.7.1 has + been removed. If you need this functionality, please file a public + issue against https://github.com/github/codeql-cli-binaries, or open + a private ticket with GitHub Support and request an escalation to + engineering. + +### New Features + +- The CodeQL CLI is now compatible with Windows 11 and Windows Server + 2022, including building databases for compiled languages. + ## Release 2.8.1 (2022-02-15) + - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM @@ -25,8 +49,8 @@ ### New Features -- Commands that find or run queries now allow you to refer to queries within a named CodeQL - pack. For example: +- Commands that find or run queries now allow you to refer to queries + within a named CodeQL pack. For example: ```sh # Analyze a database using all queries in the experimental/Security folder within the codeql/cpp-queries @@ -49,22 +73,27 @@ 'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls' ``` - The complete way to specify a set of queries is in the form `scope/name@range:path`, where: + The complete way to specify a set of queries is in the form + `scope/name@range:path`, where: - `scope/name` is the qualified name of a CodeQL pack. - - `range` is a [semver range](https://docs.npmjs.com/cli/v6/using-npm/semver#ranges). + - `range` is a [semver range][10]. - `path` is a file system path - If a `scope/name` is specified, the `range` and `path` are optional. A missing `range` - implies the latest version of the specified pack. A missing `path` implies the default - query suite of the specified pack. + If a `scope/name` is specified, the `range` and `path` are + optional. A missing `range` implies the latest version of the + specified pack. A missing `path` implies the default query suite + of the specified pack. - The `path` can be one of a `*.ql` query file, a directory containing one or more queries, or a - `.qls` query suite file. If there is no pack name specified, then a `path` must be provided, and will - be interpreted relative to the current working directory of the current process. + The `path` can be one of a `*.ql` query file, a directory + containing one or more queries, or a `.qls` query suite file. If + there is no pack name specified, then a `path` must be provided, + and will be interpreted relative to the current working directory + of the current process. - If a `scope/name` and `path` are specified, then the `path` cannot be absolute. It is considered - relative to the root of the CodeQL pack. + If a `scope/name` and `path` are specified, then the `path` cannot + be absolute. It is considered relative to the root of the CodeQL + pack. The relevant commands are: - `codeql database analyze` @@ -72,11 +101,13 @@ - `codeql execute queries` - `codeql resolve queries` + [10]: https://docs.npmjs.com/cli/v6/using-npm/semver#ranges + ### Bugs fixed - Fixed a bug that would sometimes lead to query evaluation on - M1-based Macs to crash with `Did not preallocate enough - memory` error. + M1-based Macs to crash with `Did not preallocate enough memory` + error. ## Release 2.8.0 (2022-02-04) From 14fbf78b3e06f4a300951218c0754e0211d615de Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 14 Mar 2022 12:01:01 -0700 Subject: [PATCH 056/196] Update changelog for 2.8.3 --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ef2aab..487c57f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,34 @@ you know what to do). --> +## Release 2.8.3 (2022-03-14) + +- This release of CodeQL (and all future ones) will not include the + CodeQL runner, which is now deprecated. For more information, and + instructions on how to migrate to using the CodeQL CLI, see + [CodeQL runner deprecation][11]. + + [11]: https://github.blog/changelog/2021-09-21-codeql-runner-deprecation/ + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.29) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.29 instance, you need to create them with release + 2.6.3. + +### New features + +- Executable binaries for Windows are now digitally signed by a GitHub + certificate. + +### Other changes + +- The evaluator logs produced by `--evaluator-log` now default to the + maximum verbosity level and will therefore contain more information + (and, accordingly, grow larger). The verbosity level can still be + configured with `--evaluator-log-level`. In particular, + `--evaluator-log-level=1` will restore the previous default behavior. + ## Release 2.8.2 (2022-02-28) - The bundled extractors are updated to match the versions currently From 339acad4e68b336e99cb3dced1307bb54df8e129 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Tue, 29 Mar 2022 17:34:00 +0100 Subject: [PATCH 057/196] Update changelog for 2.8.4 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 487c57f..8b41be5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ you know what to do). --> +## Release 2.8.4 (2022-03-29) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.29) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.29 instance, you need to create them with release + 2.6.3. + +### Bugs fixed + +- Fixed an error where running out of memory during query evaluation + would cause `codeql` to exit with status 34 instead of the 99 that + is documented for this condition. + +- Fixed a bug in our handling of Clang's header maps, which caused + missing files for Xcode-based projects on macOS (e.g. WebKit). + ## Release 2.8.3 (2022-03-14) - This release of CodeQL (and all future ones) will not include the From 74b59f9620b1d8a1e5f34ae602caef40c7d8efbd Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Thu, 7 Apr 2022 14:55:01 +0100 Subject: [PATCH 058/196] Update changelog for 2.8.5 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b41be5..62e176a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.8.5 (2022-04-07) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +- There are no user-facing changes in this release. + ## Release 2.8.4 (2022-03-29) - The bundled extractors are updated to match the versions currently From f81b5fdc9ab559ad3462d5b7058851c26597ba9f Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Tue, 26 Apr 2022 00:31:01 +0200 Subject: [PATCH 059/196] Update changelog for 2.9.0 --- CHANGELOG.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62e176a..1084810 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,40 @@ you know what to do). --> +## Release 2.9.0 (2022-04-26) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### New features + +- `codeql database create` now supports the `--[no-]-count-lines` + option, which was previously only available with `codeql database + init`. + +- `codeql resolve files` and `codeql database index-files` has a new + `--also-match` option, which allows users to specify glob patterns + that are applied in conjunction with the existing `--include` + option. + +### New language features + +- This release introduces experimental support for parameterized QL + modules. This language feature is still subject to change and should + not be used in production yet. + +### Bugs fixed + +- Fixed a bug that would prevent resolution of a query suite in a + published CodeQL query pack that has a reference to the pack itself. + +- Fixed inaccurate documentation of what the `--include-extension` + option to `codeql resolve files` and `codeql database index-files` + does. The actual behavior is unchanged. + ## Release 2.8.5 (2022-04-07) - The bundled extractors are updated to match the versions currently From 6d925e22496343a7bccf45a2afee046b20bf8f50 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Thu, 5 May 2022 16:55:56 +0100 Subject: [PATCH 060/196] Update changelog for 2.9.1 --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1084810..21d64fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,15 @@ you know what to do). --> +## Release 2.9.1 (2022-05-05) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + + ## Release 2.9.0 (2022-04-26) - The bundled extractors are updated to match the versions currently From 77ec3d15c63209913ec83da4bc88e440edad8ce2 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Mon, 16 May 2022 14:24:07 +0100 Subject: [PATCH 061/196] Update changelog for 2.9.2 --- CHANGELOG.md | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21d64fb..af1b917 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,7 @@ you know what to do). --> -## Release 2.9.1 (2022-05-05) +## Release 2.9.2 (2022-05-16) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of @@ -25,6 +25,44 @@ Enterprise 1.30 instance, you need to create them with release 2.7.6. +### Features removed + +- The table printed by `codeql database analyze` to summarize the + results of metric queries that were part of the analysis now reports + a single row per metric name independently of the verbosity level of + the command. Previously, at higher verbosity levels, this table + would contain multiple rows for metric names with multiple values. + +### New features + +- The tables produced by `codeql database analyze` summarizing the + results of any diagnostic and metric queries that were run now + exclude the results of queries tagged `telemetry`. + +- Uploading SARIF results using the `codeql github upload-results` + command now has a timeout of 5 minutes. + +- Downloading CodeQL packs using the `codeql pack download`, + `codeql pack install` and related commands now have a timeout of + 5 minutes and will retry 3 times before failing. Similar behavior + has been added to the `codeql pack publish` command. + +- The `codeql generate log-summary` command will now print progress + updates to `stderr`. + +### Bugs fixed + +- Fixed a bug that could make it unpredictable whether the QL compiler + reports problems about query metadata tags, and thereby make `codeql + test run` fail spuriously in some cases. + +## Release 2.9.1 (2022-05-05) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. ## Release 2.9.0 (2022-04-26) From 2352f17671cfc10dfbb315e08d6691df7a23e47f Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Tue, 31 May 2022 09:36:37 -0700 Subject: [PATCH 062/196] Update changelog for 2.9.3 --- CHANGELOG.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index af1b917..d7b786e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,51 @@ you know what to do). --> +## Release 2.9.3 (2022-05-31) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### New features + +- Users can now use CodeQL Packaging Beta to publish and download + CodeQL packs on GitHub Enterprise Server (GHES) versions 3.6 and + later. + + To authenticate to a package registry on GHES 3.6+, first create a + `~/.codeql/qlconfig.yml` file. For example, the following file + specifies that all CodeQL packages should be uploaded to the GHES + instance with the hostname `GHE_HOSTNAME`: + + ```yml + registries: + - packages: '*' + url: https://containers.GHE_HOSTNAME/v2/ + ``` + + You can now download public packages from GHES using + `codeql pack download`. + + To publish any package or download private packages, authenticate to + GHES by specifying registry/token pairs in the + `CODEQL_REGISTRIES_AUTH` environment variable. You can authenticate + using either a GitHub Apps token or a personal access token. For + example, + `https://containers.GHEHOSTNAME1/v2/=TOKEN1,https://containers.GHEHOSTNAME2/v2/=TOKEN2` + will authenticate the CLI to the `GHEHOSTNAME1` and `GHEHOSTNAME2` + GHES instances. + +### Bugs Fixed + +- Fixed a bug where precompiled CodeQL packages in the CodeQL bundle were + being recompiled if they were in a read-only directory. + +- Fixed a bug where new versions of the VS Code extension wouldn't run two + queries in parallel against one database. + ## Release 2.9.2 (2022-05-16) - The bundled extractors are updated to match the versions currently From 4ff0d7953bf028e7c078f38a19f20f5a35dd53fb Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Mon, 20 Jun 2022 12:34:45 +0100 Subject: [PATCH 063/196] Update changelog for 2.9.4 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d7b786e..c375184 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ you know what to do). --> +## Release 2.9.4 (2022-06-20) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### New features + +- Users of CodeQL Packaging Beta can now optionally authenticate to + Container registries on GitHub Enterprise Server (GHES) versions 3.6 + and later using standard input instead of the `CODEQL_REGISTRIES_AUTH` + environment variable. To authenticate via standard input, pass + `--registries-auth-stdin`. The value you provide will override the + value of the `CODEQL_REGISTRIES_AUTH` environment variable. + ## Release 2.9.3 (2022-05-31) - The bundled extractors are updated to match the versions currently From 98c20a4dbd1f3ebedcc574b1d1d4532540404525 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 27 Jun 2022 10:24:01 -0700 Subject: [PATCH 064/196] Update changelog for 2.10.0 --- CHANGELOG.md | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c375184..f93f76f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,68 @@ you know what to do). --> +## Release 2.10.0 (2022-06-27) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### Breaking changes + +- The `--format=stats` option of `codeql generate log-summary` has been + renamed to `--format=overall`. It now produces a richer JSON object + that, in addition to the previous statistics about the run (which can + be found in the `stats` property) also records the most expensive + predicates in the evaluation run. + +### Potentially breaking changes + +- The `codeql resolve ml-model` command now requires one or more query + specifications as command line arguments in order to determine the set + of starting packs from which to initiate the resolution process. The + command will locate all ML models in any qlpack that is a transitive + dependency of any of the starting packs. Also, the output of the + command has been expanded to include for each model the containing + package's name, version, and path. + +- The `buildMetadata` inside of compiled CodeQL packs no longer contains + a `creationTime` property. This was removed in order to ensure that + the content of a CodeQL pack is identical when it is re-compiled. + +- The `codeql pack download` command, when used with the `--dir` option, + now downloads requested packs in directories corresponding to their + version numbers. Previously, + `codeql pack download --dir ./somewhere codeql/java-queries@0.1.2` + would download the pack into the `./somewhere/codeql/java-queries` + directory. Now, it will download the pack into the + `./somewhere/codeql/java-queries/0.1.2` directory. This allows you to + download multiple versions of the same pack using a single command. + +### New features + +- You can now include diagnostic messages in the summary produced by + the `--print-diagnostics-summary` option of the + `codeql database interpret-results` and `codeql database analyze` + commands by running these commands at high verbosity levels. + +### Bugs fixed + +- Fixed a bug where `codeql pack download`, when used with the `--dir` + option, would not download a pack that is in the global package cache. + +- Fixed a bug where some versions of a CodeQL package could not be + downloaded if there are more than 100 versions of this package in the + package registry. + +- Fixed a bug where the `--also-match` option for `codeql resolve files` + and `codeql database index-files` does not work with relative paths. + +- Fixed a bug that caused `codeql query decompile` to ignore the + `--output` option when producing bytecode output (`--kind=bytecode`), + writing only to `stdout`. + ## Release 2.9.4 (2022-06-20) - The bundled extractors are updated to match the versions currently From 6998e5c83bdce05eb4b50bd17a0ed576aa4efdd8 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 19 Jul 2022 14:46:01 +0100 Subject: [PATCH 065/196] Update changelog for 2.10.1 --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f93f76f..0e43b97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,19 @@ you know what to do). --> +## Release 2.10.1 (2022-07-19) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### New features + +- Improved error message from `codeql database analyze` when a query is + missing `@id` or `@kind` query metadata. + ## Release 2.10.0 (2022-06-27) - The bundled extractors are updated to match the versions currently From 483a38562eb5243cba4df7d756ffe252585b1d00 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Tue, 2 Aug 2022 09:30:49 -0700 Subject: [PATCH 066/196] Update changelog for 2.10.2 --- CHANGELOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e43b97..e4f1c68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,49 @@ you know what to do). --> +## Release 2.10.2 (2022-08-02) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### Breaking change + +- The option `--compiler-spec` to `codeql database create` (and + `codeql database trace-command`) no longer works. It is replaced by + `--extra-tracing-config`, which accepts a tracer configuration file + in the new, Lua-based tracer configuration format instead. See + `tools/tracer/base.lua` for the precise API available. If you need + help help porting your existing compiler specification files, please + file a public issue in https://github.com/github/codeql-cli-binaries, + or open a private ticket with GitHub support and request an + escalation to engineering. + +### Potentially breaking changes + +- Versions of the CodeQL extension for Visual Studio Code released + before February 2021 may not work correctly with this CLI, in + particular if database upgrades are necessary. We recommend keeping + your VS Code extension up-to-date. + +### Deprecation + +- The experimental `codeql resolve ml-models` command has been + deprecated. Advanced users calling this command should use the new + `codeql resolve extensions` command instead. + +### New features + +- The `codeql github upload-sarif` command now supports a `--merge` + option. If this option is provided, the command will accept the paths + to multiple SARIF files, and will merge those files before uploading + them as a single analysis. This option is recommended _only_ for + backwards compatibility with old analyses produced by the CodeQL + Runner, which combined the results for multiple languages into a + single analysis. + ## Release 2.10.1 (2022-07-19) - The bundled extractors are updated to match the versions currently From d449a263c2413f220149dfb58db5a2beccb45f7f Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Mon, 15 Aug 2022 16:57:32 +0100 Subject: [PATCH 067/196] Update changelog for 2.10.3 --- CHANGELOG.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4f1c68..77fcd54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,27 @@ you know what to do). --> +## Release 2.10.3 (2022-08-15) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +### New features + +- When called with `--start-tracing`, the `codeql database init` command + now accepts extractor options for the indirect tracing environment via + `--extractor-option`. Users should continue to specify extractor options + for direct tracing environments by passing them to + `codeql database trace-command` invocations. + +### Other changes + +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.4. + ## Release 2.10.2 (2022-08-02) - The bundled extractors are updated to match the versions currently From a0418f108e56ffde70d1bd5ce38ca67d33260e1b Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 19 Aug 2022 13:24:19 -0700 Subject: [PATCH 068/196] Use correct CLI command name --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 77fcd54..0f2ad55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -73,7 +73,7 @@ ### New features -- The `codeql github upload-sarif` command now supports a `--merge` +- The `codeql github upload-results` command now supports a `--merge` option. If this option is provided, the command will accept the paths to multiple SARIF files, and will merge those files before uploading them as a single analysis. This option is recommended _only_ for From c627a42c3a8862b72c68aa10e95943ac4b7ac515 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Wed, 31 Aug 2022 13:10:36 +0100 Subject: [PATCH 069/196] Update CHANGELOG for 2.10.4 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f2ad55..d2e8962 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.10.4 (2022-08-31) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +- This release does not include any user-facing changes. + ## Release 2.10.3 (2022-08-15) - The bundled extractors are updated to match the versions currently From 355609ae9671ab583ccf23e9136688296339ae94 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 13 Sep 2022 14:44:46 +0100 Subject: [PATCH 070/196] Update changelog for 2.10.5 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d2e8962..86a4e2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.10.5 (2022-09-13) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.30) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.30 instance, you need to create them with release + 2.7.6. + +- This release does not include any user-facing changes. + ## Release 2.10.4 (2022-08-31) - The bundled extractors are updated to match the versions currently From b8161ac382904fa98ef0867b761754f686b4cfb9 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 13 Sep 2022 15:09:27 +0100 Subject: [PATCH 071/196] Update changelog for 2.10.5 --- CHANGELOG.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 86a4e2a..509e1b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,12 +19,6 @@ ## Release 2.10.5 (2022-09-13) -- The bundled extractors are updated to match the versions currently - used on LGTM.com. These are newer than the last release (1.30) of - LGTM Enterprise. If you plan to upload databases to an LGTM - Enterprise 1.30 instance, you need to create them with release - 2.7.6. - - This release does not include any user-facing changes. ## Release 2.10.4 (2022-08-31) From dd3de9d4fd4c779c6d76b833a712933bb3348e8d Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Thu, 15 Sep 2022 09:48:52 -0700 Subject: [PATCH 072/196] Update CHANGELOG.md for rc/3.7 --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 509e1b1..32c2820 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,11 @@ ## Release 2.10.5 (2022-09-13) -- This release does not include any user-facing changes. +### New features + +- You can now define which registries should be used for downloading and publishing CodeQL packs on a + per-workspace basis by creating a `codeql-workspace.yml` file and addinga `registries` block. For + more infomation, see [About CodeQL Workspaces](https://codeql.github.com/docs/codeql-cli/about-codeql-workspaces/). ## Release 2.10.4 (2022-08-31) From 54039b9444169a61be1efceb86cf72a40cd84e34 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Thu, 15 Sep 2022 09:53:33 -0700 Subject: [PATCH 073/196] Fix typo --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 32c2820..8fa47c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ ### New features - You can now define which registries should be used for downloading and publishing CodeQL packs on a - per-workspace basis by creating a `codeql-workspace.yml` file and addinga `registries` block. For + per-workspace basis by creating a `codeql-workspace.yml` file and adding a `registries` block. For more infomation, see [About CodeQL Workspaces](https://codeql.github.com/docs/codeql-cli/about-codeql-workspaces/). ## Release 2.10.4 (2022-08-31) From 11069f90196335d939d8815f9647d4474bf4df5b Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Wed, 28 Sep 2022 09:55:14 -0700 Subject: [PATCH 074/196] Update changelog for 2.11.0 --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8fa47c5..0f61287 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ you know what to do). --> +## Release 2.11.0 (2022-09-28) + +### Deprecation + +- The CodeQL CLI now uses Python 3 to extract both Python 2 and Python 3 + databases. Correspondingly, support for using Python 2 to extract + Python databases is now deprecated. Starting with version 2.11.3, you + will need to install Python 3 to extract Python databases. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.4. + ## Release 2.10.5 (2022-09-13) ### New features From 7471af16a61abf3c1f3b63be0919fb4f63e0912b Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 11 Oct 2022 18:27:21 +0100 Subject: [PATCH 075/196] Update changelog for 2.11.1 --- CHANGELOG.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f61287..9d821d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,45 @@ you know what to do). --> +## Release 2.11.1 (2022-10-11) + +## Breaking changes + +- Pack installation using the CodeQL Packaging beta will now fail if a + compatible version cannot be found. This replaces the previous + behavior where `codeql pack download` and related commands would + instead install the latest version of the pack in this situation. + +## Deprecations + +- The `--[no-]count-lines` option to `codeql database create` and + related commands is now deprecated and will be removed in a future + release of the CodeQL CLI (earliest 2.12.0). It is replaced by + `--[no-]calculate-baseline` to reflect the additional baseline + information that is now captured as of this release. + +### New features + +- Subcommands that compile QL accept a new `--no-release-compatibility` + option. It does nothing for now, but in the future it will be used + to control a trade-off between query performance and compatibility + with older/newer releases of the QL evaluator. +- `codeql database analyze` and related commands now support absolute + paths containing the `@` or `:` characters when specifying which queries + to run. To reference a query file, directory, or suite whose path contains + a literal `@` or `:`, prefix the query specifier with `path:`, for example: + ```shell + codeql database analyze --format=sarif-latest --output=results path:C:/Users/ci/workspace@2/security/query.ql + ``` +### Bugs fixed + +- It is no longer an error to call `codeql pack create ` with a `` + option pointing to a file name. The CLI will walk up the directory tree and + run the command in the first directory containing the `qlpack.yml` or `codeql-pack.yml` file. +- Fixed a concurrency error observed when using `codeql database import` or + `codeql database finalize` with multiple threads and multiple additional + databases on a C++ codebase. + ## Release 2.11.0 (2022-09-28) ### Deprecation From d9ff04b743ea34b48f4057ab6d6624a94678f177 Mon Sep 17 00:00:00 2001 From: Edoardo Pirovano Date: Tue, 25 Oct 2022 09:40:49 +0100 Subject: [PATCH 076/196] Update changelog for 2.11.2 --- CHANGELOG.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d821d2..d3ec078 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,16 +17,68 @@ you know what to do). --> +## Release 2.11.2 (2022-10-25) + +### Breaking changes + +- Bundling and publishing a CodeQL pack will no longer include nested + CodeQL packs. If you want to include a nested pack in your published pack, + then you must explicitly include it using the `include` property in the + top-level `qlpack.yml` file. + + For example, if your package structure looks like this: + + ```text + qlpack.yml + nested-pack + ∟ qlpack.yml + query.ql + ``` + + then the contents of `nested-pack` will not be included by default within + the published package. To include `nested-pack`, add an entry like this + to the top level `qlpack.yml` file: + + ```yml + include: + - nested-pack/** + ``` + +### Bugs fixed + +- Using the `--codescanning-config=` option in + `codeql database init` will now correctly process the `paths` and + `pathsIgnore` properties of the configuration file in a way that is + identical to the behavior of the `codeql-action`. Previously, `paths` + or `pathsIgnore` entries that end in `/**` or start with `/` were + incorrectly rejected by the CLI. + +- Fixed a bug where the `--compilation-cache` option to + `codeql pack publish` and `codeql pack create` was being ignored + when creating a query pack. Now, the indicated cache is used + when pre-compiling the queries in it. + +- Fixed a bug that would make the "Show DIL" command in the VSCode + extension display nothing. + +### Other changes + +- Emit a detailed warning if package resolution fails, the legacy + `--search-path` option is provided, _and_ there is at least one + referenced pack that does not use legacy package resolution. + In this case, `--additional-packs` should be used to extend the + search to additional directories, instead of `--search-path`. + ## Release 2.11.1 (2022-10-11) -## Breaking changes +### Breaking changes - Pack installation using the CodeQL Packaging beta will now fail if a compatible version cannot be found. This replaces the previous behavior where `codeql pack download` and related commands would instead install the latest version of the pack in this situation. -## Deprecations +### Deprecations - The `--[no-]count-lines` option to `codeql database create` and related commands is now deprecated and will be removed in a future From 9223ad3ff735947c3393f14b974e63149bf04660 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Fri, 11 Nov 2022 12:17:08 +0000 Subject: [PATCH 077/196] Update changelog for 2.11.3 --- CHANGELOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3ec078..0362ffc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,49 @@ you know what to do). --> +## Release 2.11.3 (2022-11-11) + +### Breaking changes + +- The `codeql pack ls --format json` deep plumbing command now returns + only the `name` and `version` properties for each found pack. + +### Potentially breaking changes + +- `codeql pack download`, `codeql pack install`, and `codeql pack add` + will ignore CodeQL packs with pre-release versions, unless the + `--allow-prerelease` option is passed to the command. This brings + these commands into alignment with `codeql pack publish` that will + avoid publishing CodeQL packs with pre-release versions unless the + `--allow-prerelease` option is specified. Pre-release versions have + the following format: `X.Y.Z-qualifier` where `X`, `Y`, and `Z` are + respectively the major, minor, and patch number. `qualifier` is the + pre-release version. For more information about pre-releases, see + the + [Semantic Versioning specification](https://semver.org/#spec-item-9). + +### Deprecations + +- The `--[no-]fast-compilation` option to `codeql query compile` is + now deprecated. + +### New features + +- `codeql resolve files` and `codeql database index-files` have a new + `--find-any` option, which finds at most one match. + +### Miscellaneous + +- The build of Apache Commons Text that is bundled with the CodeQL CLI + has been updated to version 1.10.0. While previous releases shipped + with version 1.6 of the library, no part of the CodeQL CLI + references the `StringSubstitutor` class that the recently disclosed + [CVE-2022-42889](https://github.com/advisories/GHSA-599f-7c49-w659) + vulnerability applies to. We therefore do not believe that running + previous releases of CodeQL exposes users to this vulnerability. +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.5. + ## Release 2.11.2 (2022-10-25) ### Breaking changes From 3dc36ae6cf9d20234a8fe9865eb2f61d648b814f Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Thu, 24 Nov 2022 14:02:19 +0000 Subject: [PATCH 078/196] Update changelog for 2.11.4 --- CHANGELOG.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0362ffc..5b5dee2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,30 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.11.4 (2022-11-24) + +### New features + +- Kotlin support is now in beta. This means that Java analyses will also + include Kotlin code by default. Kotlin support can be disabled by + setting `CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN` to `true` in + the environment. + +### Potentially breaking changes + +- CodeQL 2.11.1 to 2.11.3 contained a bug in [indirect build tracing](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#using-indirect-build-tracing) on Windows + when using `codeql database init` with the [`--trace-process-level`](https://codeql.github.com/docs/codeql-cli/manual/database-init/#cmdoption-codeql-database-init-trace-process-level) flag. + In these versions, when `--trace-process-level` was set to a value greater than zero, + (or left at the default value of 1), CodeQL attempted to inject its build tracer + at a higher level in the process tree than the requested process level. + This could lead to errors of the form "No source code found" or + "Process tree ended before reaching required level". + From 2.11.4 onwards, the CodeQL build tracer is injected at the requested process level. + +### Deprecations + +- The `--[no-]fast-compilation` option to `codeql test run` is now + deprecated. ## Release 2.11.3 (2022-11-11) From 441d966d167c8bbc88f094484490e10d9168c886 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Wed, 7 Dec 2022 10:51:57 -0800 Subject: [PATCH 079/196] Update changelog for 2.11.5 --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b5dee2..edb6bda 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,17 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.11.5 (2022-12-07) + +### Bugs Fixed + +- Fixed a bug where it was not possible to run queries in CodeQL query packs for C# + that use the legacy `libraryPathDependencies` property in their `qlpack.yml` file. + The associated error message complained about undefined extensional predicates. + +- Fixed a bug that could cause log summary generation to fail in vscode. + ## Release 2.11.4 (2022-11-24) ### New features From 6221102ae364e21f07ffe1245dac79c7315f11d7 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Fri, 9 Dec 2022 11:18:04 -0800 Subject: [PATCH 080/196] Remove changelog entry The bug fix did not make it into this release. --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index edb6bda..17cc5f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,10 +21,6 @@ ### Bugs Fixed -- Fixed a bug where it was not possible to run queries in CodeQL query packs for C# - that use the legacy `libraryPathDependencies` property in their `qlpack.yml` file. - The associated error message complained about undefined extensional predicates. - - Fixed a bug that could cause log summary generation to fail in vscode. ## Release 2.11.4 (2022-11-24) From 631e69b2fd5ce08dda6228c200ad3721188dd57e Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 13 Dec 2022 13:12:39 +0000 Subject: [PATCH 081/196] Update changelog for 2.11.6 --- CHANGELOG.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 17cc5f3..3e40a40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,27 @@ you know what to do). --> +## Release 2.11.6 (2022-12-13) + +### Breaking changes + +- Java and Kotlin analyses in this release of the CLI and all earlier + releases are incompatible with Kotlin 1.7.30 and later. To prevent + code scanning alerts being spuriously dismissed, Java and Kotlin + analyses will now fail when using Kotlin 1.7.30 or later. + + If you are unable to use Kotlin 1.7.29 or earlier, you can disable + Kotlin support by setting + `CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN` to `true` in the + environment. + +### Bugs fixed + +- Fixed a bug where it was not possible to run queries in CodeQL query + packs for C# that use the legacy `libraryPathDependencies` property + in their `qlpack.yml` file. The associated error message complained + about undefined extensional predicates. + ## Release 2.11.5 (2022-12-07) ### Bugs Fixed From 6d750b5e30a801978a237301a396809cd9334668 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Tue, 10 Jan 2023 15:43:06 -0500 Subject: [PATCH 082/196] Update changelog for 2.12.0 --- CHANGELOG.md | 65 ++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e40a40..0fce428 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,61 @@ you know what to do). --> +## Release 2.12.0 (2023-01-10) + +### Breaking changes + +- The `--[no-]count-lines` option to `codeql database create` and related commands that was + deprecated in 2.11.1 has been removed. Users of this option should instead pass + `--[no-]calculate-baseline`. + +### New features + +- Query packs created by `codeql pack create`, `codeql pack bundle`, and `codeql pack release` now + contain precompiled queries in a new format that aims to be compatible with future (and, to a + certain extent, past) releases of the CodeQL CLI. Previously the precompiled queries were in a + format specific to each CLI release, and all other releases would need to re-compile queries. + + Published packs contain precompiled queries in files with a `.qlx` extension located next to each + query's `.ql` source file. In case of differences between the `.ql` and `.qlx` files, the `.qlx` + file takes priority when evaluating queries from the command line, so if you need to modify a + published pack, be sure to delete the `.qlx` files first. + + A new `--precompile` flag to `codeql query compile` can be used to construct `*.qlx` file + explicitly, but in all usual cases it should be enough to rely on `codeql pack create` doing the + right thing. +- The `codeql database init` command now accepts a PAT that allows you to download queries from + external, private repositories when using the `--codescanning-config ` option. For + example, you can specify the following queries block in the config file, which will checkout the main + branch of the `codeql-test/my-private-repository` repository and evaluate any queries found in that + repository: + + ```yaml + queries: + - codeql-test/my-private-repository@main + ``` + + If the repository is private, you can add a `--external-repository-token-stdin` option and supply a + PAT with appropriate permissions via standard input. For more information on queries and external + repositories in Code Scanning, see [Using queries in QL packs](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs). +- The baseline information produced by `codeql database init` and + `codeql database create` now accounts for + [`paths` and `paths-ignore` configuration](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#specifying-directories-to-scan). +- In the VS Code extension, recursive calls will be marked with inlay + hints. These can be disabled with the global inlay hints setting + (`editor.inlayHints.enabled`). If you just want to disable them for + codeql the settings can be scoped to just codeql files (language id is `ql`). + See [Language Specific Editor Settings](https://code.visualstudio.com/docs/getstarted/settings#_language-specific-editor-settings) + in the VS Code documentation for more information. +- The CLI now gives a more helpful error message when asked to run queries on a + database that has not been finalized. + +### Bugs fixed + +- Fixed a bug where the `codeql pack install` command would fail if + a [CodeQL configuration file](https://codeql.github.com/docs/codeql-cli/specifying-command-options-in-a-codeql-configuration-file/#using-a-codeql-configuration-file) + is used and the `--additional-packs` option is specified. + ## Release 2.11.6 (2022-12-13) ### Breaking changes @@ -52,7 +107,7 @@ include Kotlin code by default. Kotlin support can be disabled by setting `CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN` to `true` in the environment. - + ### Potentially breaking changes - CodeQL 2.11.1 to 2.11.3 contained a bug in [indirect build tracing](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#using-indirect-build-tracing) on Windows @@ -187,9 +242,9 @@ option. It does nothing for now, but in the future it will be used to control a trade-off between query performance and compatibility with older/newer releases of the QL evaluator. -- `codeql database analyze` and related commands now support absolute - paths containing the `@` or `:` characters when specifying which queries - to run. To reference a query file, directory, or suite whose path contains +- `codeql database analyze` and related commands now support absolute + paths containing the `@` or `:` characters when specifying which queries + to run. To reference a query file, directory, or suite whose path contains a literal `@` or `:`, prefix the query specifier with `path:`, for example: ```shell codeql database analyze --format=sarif-latest --output=results path:C:/Users/ci/workspace@2/security/query.ql @@ -197,7 +252,7 @@ ### Bugs fixed - It is no longer an error to call `codeql pack create ` with a `` - option pointing to a file name. The CLI will walk up the directory tree and + option pointing to a file name. The CLI will walk up the directory tree and run the command in the first directory containing the `qlpack.yml` or `codeql-pack.yml` file. - Fixed a concurrency error observed when using `codeql database import` or `codeql database finalize` with multiple threads and multiple additional From 970025b241343efdd9b81974134d46bdec17d0bb Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Mon, 23 Jan 2023 17:49:11 +0000 Subject: [PATCH 083/196] Update changelog for 2.12.1 --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0fce428..3974372 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,15 @@ you know what to do). --> +## Release 2.12.1 (2023-01-23) + +### New features + + - Added a new command-line flag `--expect-discarded-cache`, which gives a hint + to the evaluator that the evaluation cache will be discarded after analysis + completes. This allows it to avoid some unnecessary writes to the cache, for + predicates that aren't needed by the query/suite being evaluated. + ## Release 2.12.0 (2023-01-10) ### Breaking changes From e7708ebb980165764b60fa4b214276ac2bd4121a Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Tue, 7 Feb 2023 11:14:59 -0800 Subject: [PATCH 084/196] Update changelog for 2.12.2 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3974372..1af9345 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ you know what to do). --> +## Release 2.12.2 (2023-02-07) + +### Bugs fixed + +- Fixed a QL evaluator bug introduced in release 2.12.1 which could in + certain rare cases lead to wrong analysis results. + +- Fixed handling of `-Xclang ` arguments passed to the `clang` compiler which + could cause missing extractions for C++ code bases. + +- Fixed a bug where the `--overwrite` option was failing for database clusters. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.6. + ## Release 2.12.1 (2023-01-23) ### New features From fb60d01aadddd9abd108bfd23272919c2792943b Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Thu, 23 Feb 2023 15:49:28 +0000 Subject: [PATCH 085/196] Update changelog for 2.12.3 --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1af9345..2c54c64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ you know what to do). --> +## Release 2.12.3 (2023-02-23) + +### New features + +- The CodeQL compiler now produces better error messages when it is unable + to find a QL library that the query being evaluated depends on. + +### Bugs fixed + +- Fixed a bug where the CLI would refuse to complete database creation + if the OS reports less than about 1.5 GB of physical memory. Now an + attempt will be made even on low-memory systems (but it might still + run out of memory unless there's swap space available). + ## Release 2.12.2 (2023-02-07) ### Bugs fixed From 070353407ccac9159cda9607991dcd8d8c0416c1 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 9 Mar 2023 10:29:24 -0500 Subject: [PATCH 086/196] Update changelog for 2.11.4 --- CHANGELOG.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c54c64..4393f35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,43 @@ you know what to do). --> +## Release 2.12.4 (2023-03-09) + +### Breaking changes + +- The default value of the `--mode` switch to `codeql pack install` has changed. The default is now `--mode minimal-update`. + Previously, it was `use-lock`. + +### New features + +- The per-pack compilation cache has been replaced with a global compilation cache + found within `~/.codeql`. +- `codeql pack install` now uses a new algorithm to determine which versions of + the pack's dependencies to use, based on the [PubGrub](https://nex3.medium.com/pubgrub-2fb6470504f) + algorithm. The new algorithm is able to find a solution for many cases that + the previous algorithm would fail to solve. When the new algorithm is unable + to find a valid solution, it generates a detailed error message explaining + why there is no valid solution. +- Added a new command, `codeql pack upgrade`. This command is similar to `codeql pack install`, + except that it ignores any existing lock file, installs the latest compatible version of each + dependency, and writes a new lock file. This is equivalent to `codeql pack install --mode update`. + Note that the `--mode` switch to `codeql pack install` is now deprecated. +- Added a new command, `codeql pack ci`. This command is similar to `codeql pack install`, + except if the existing lock file is missing, or if it conflicts with the version constraints in + the `qlpack.yml` file, the command generates an error. This is equivalent to + `codeql pack install --mode verify`. Note that the `--mode` switch to `codeql pack install` is now + deprecated. + +### Deprecations + +- The `--freeze` switch for `codeql pack create`, `codeql pack bundle`, and `codeql pack publish` + is now deprecated and ignored, as there is no longer a cache within a pack. +- The `--mode update` switch to `codeql pack resolve-dependencies` is now deprecated. Instead, use + the new `--mode upgrade` switch, which has identical behavior. +- The `--mode` switch to `codeql pack install` is now deprecated. + - Instead of `--mode update`, use `codeql pack upgrade`. + - Instead of `--mode verify`, use `codeql pack ci`. + ## Release 2.12.3 (2023-02-23) ### New features From 0feb390a8549fffec43f6181980a887a011843ca Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 21 Mar 2023 15:06:48 +0000 Subject: [PATCH 087/196] Update changelog for 2.12.5 --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4393f35..bab8657 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,29 @@ you know what to do). --> +## Release 2.12.5 (2023-03-21) + +### New features + +- The `codeql pack install` command now accepts a `--additional-packs` + option. This option takes a list of directories to search for + locally available packs when resolving which packs to install. Any + pack that is found locally through `--additional-packs` will + override any other version of a pack found in the package registry. + Locally resolved packs are not added to the lock file. + + Because the use of `--additional-packs` when running + `codeql pack install` makes running queries dependent on the local + state of the machine initially invoking `codeql pack install`, a + warning is emitted if any pack is found outside of the package + registry. This warning can be suppressed by using the + `--no-strict-mode` option. + +### Bugs fixed + +- Fix a bug in `codeql query run` where queries whose path contain + colons cannot be run. + ## Release 2.12.4 (2023-03-09) ### Breaking changes From 37f85486c0e3a6a50fa2af6ce5ba81b7d88fa3ab Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 4 Apr 2023 16:03:41 +0100 Subject: [PATCH 088/196] Add 2.12.6 to the CHANGELOG --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bab8657..58a655c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,29 @@ you know what to do). --> +## Release 2.12.6 (2023-04-04) + +### New features + +- Several experimental subcommands have been added in support of the + new [code scanning tool status page](https://github.blog/changelog/2023-03-28-code-scanning-shows-the-health-of-tools-enabled-on-a-repository/). + These include `codeql database add-diagnostic`, + `codeql database export-diagnostics`, and the + `codeql diagnostic add` and `codeql diagnostic export` plumbing + subcommands. + +### Bugs fixed + +- Fixed a bug in `codeql database analyze` and related commands + where the `--max-paths` option was not respected correctly when + multiple alerts with the same primary code location were grouped together. + (This grouping is the default behavior unless the `--no-group-alerts` + option is passed.) + This bug caused some SARIF files produced by CodeQL to exceed the limits + on the number of paths (`threadFlows`) accepted by code scanning, + leading to errors when uploading results. + + ## Release 2.12.5 (2023-03-21) ### New features From 20c35f9370edec6d6a3dde10fdfb17885e8d2d0f Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 18 Apr 2023 21:16:29 +0100 Subject: [PATCH 089/196] Update changelog for 2.12.7 --- CHANGELOG.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58a655c..8e10b9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,8 +17,33 @@ you know what to do). --> +## Release 2.12.7 (2023-04-18) + +### Bugs fixed + +- Fixed a bug in `codeql database upload-results` where the subcommand + would fail with "A fatal error occurred: Invalid SARIF.", reporting + an `InvalidDefinitionException`. This issue occurred when the SARIF + file contained certain kinds of diagnostic information. + ## Release 2.12.6 (2023-04-04) +### Known issues + +- We recommend that customers using the CodeQL CLI in a third party CI + system do not upgrade to this release, due to an issue with `codeql + github upload-results`. Instead, please use CodeQL 2.12.5, or, when + available, CodeQL 2.12.7 or 2.13.1. + + This issue occurs when uploading certain kinds of diagnostic information and + causes the subcommand to fail with "A fatal error occurred: Invalid + SARIF.", reporting an `InvalidDefinitionException`. + + Customers who wish to use CodeQL 2.12.6 or 2.13.0 can + work around the problem by passing `--no-sarif-include-diagnostics` + to any invocations of `codeql database analyze` or `codeql database + interpret-results`. + ### New features - Several experimental subcommands have been added in support of the From 20e24f86589e0753319a4317d3e0ca1d3fdec693 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 20 Apr 2023 07:44:26 -0700 Subject: [PATCH 090/196] Update changelog for 2.13.0 --- CHANGELOG.md | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e10b9c..eb2cfea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,73 @@ you know what to do). --> +## Release 2.13.0 (2023-04-20) + +### Known issues + +- We recommend that customers using the CodeQL CLI in a third party CI + system do not upgrade to this release, due to an issue with `codeql + github upload-results`. Instead, please use CodeQL 2.12.5, or, when + available, CodeQL 2.12.7 or 2.13.1. For more information, see the + "Known issues" section for CodeQL 2.12.6. + +### Potentially breaking changes + +- In `codeql pack add`, the dependency that is added to the `qlpack.yml` file will now allow any + version of the pack that is compatible with the specified version (`^version`) in the following + cases: + - When no version is specified (`codeql pack add codeql/cpp-all`). + - When the version is specified as `latest` (`codeql pack add codeql/cpp-all@latest`). + - When a single version is specified (`codeql pack add codeql/cpp-all@1.0.0`). + + The `^version` dependency allows any version of that pack with no breaking changes since `version`. + For example, `^1.2.3` would allow versions `1.2.3`, `1.2.5`, and `1.4.0`, but not `2.0.0`, because + changing the major version number to `2` indicates a breaking change. + + Using `^version` ensures that the added pack is not needlessly constrained to an exact version by default. + +- Upper-case variable names are no longer accepted by the QL compiler. + + Such variable names have produced a deprecation warning since + release 2.9.2 (released 2022-05-16), so QL code that compiles + without warnings with a recent release of the CLI should still work. + +### New features + +- `codeql database analyze` and related commands now export file + coverage information by default. GHAS customers using CodeQL in + third-party CI systems will now see file coverage information on the + [tool status page](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page) + without needing to modify their CI workflows. + +### Deprecations + +- The possibility to omit `override` annotations on class member + predicates that override a base class predicate has been deprecated. + This is to avoid confusion with shadowing behaviour in the + presence of final member predicates. + + ```ql + class Foo extends Base { + final predicate foo() { ... } + + predicate bar() { ... } + + predicate baz() { ... } + } + + class Bar extends Foo { + // This method shadows Foo::foo. + predicate foo() { ... } + + // This used to override Foo::bar with a warning, is now deprecated. + predicate bar() { ... } + + // This correctly overrides Foo::baz + override predicate baz() { ... } + } + ``` + ## Release 2.12.7 (2023-04-18) ### Bugs fixed @@ -64,7 +131,6 @@ on the number of paths (`threadFlows`) accepted by code scanning, leading to errors when uploading results. - ## Release 2.12.5 (2023-03-21) ### New features From 163b35080c8884d96a09574b214d61ea5c8d3e4b Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 2 May 2023 07:15:52 -0700 Subject: [PATCH 091/196] Update README.md Add information about where to raise issues. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 4617044..a4fe449 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ This repo holds binaries for the CodeQL CLI. for placing it in a location where the CLI can find it. 4. Read the rest of the [CodeQL CLI documentation](https://codeql.github.com/docs/codeql-cli/). +Found a bug or have a wuestion? please raise an issue in the [github/codeql](https://github.com/github/codeq) repository. + # License By downloading, you agree to the [GitHub CodeQL Terms & Conditions](https://securitylab.github.com/tools/codeql/license/). From 0a069410f1a9d0ccb9abcf137de210ca90ebab66 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 2 May 2023 07:22:46 -0700 Subject: [PATCH 092/196] Update README.md Co-authored-by: Henning Makholm --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a4fe449..4d67e38 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ This repo holds binaries for the CodeQL CLI. for placing it in a location where the CLI can find it. 4. Read the rest of the [CodeQL CLI documentation](https://codeql.github.com/docs/codeql-cli/). -Found a bug or have a wuestion? please raise an issue in the [github/codeql](https://github.com/github/codeq) repository. +Found a bug or have a question? please raise an issue in the [github/codeql](https://github.com/github/codeq) repository. # License From f78a5006c2e2d1a662310f3953cfc2443583c8d1 Mon Sep 17 00:00:00 2001 From: Andrew Eisenberg Date: Tue, 2 May 2023 09:14:17 -0700 Subject: [PATCH 093/196] Use more precise issue link --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4d67e38..161270f 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,9 @@ This repo holds binaries for the CodeQL CLI. for placing it in a location where the CLI can find it. 4. Read the rest of the [CodeQL CLI documentation](https://codeql.github.com/docs/codeql-cli/). -Found a bug or have a question? please raise an issue in the [github/codeql](https://github.com/github/codeq) repository. +# Found a bug or have a question? + +Please raise an issue in the [github/codeql](https://github.com/github/codeql/issues/new/choose) repository. # License From 2e6a3db7ffea3c9fa80ffd40da22b3ded8ad19ac Mon Sep 17 00:00:00 2001 From: Aditya Sharad <6874315+adityasharad@users.noreply.github.com> Date: Tue, 2 May 2023 15:22:07 -0700 Subject: [PATCH 094/196] Create issue configuration directing to the github/codeql repo Encourage users to create issues in `github/codeql`. Follows the format at https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository#configuring-the-template-chooser. --- .github/ISSUE_TEMPLATE/config.yml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/config.yml diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..27754bb --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: false +contact_links: + - name: Report an issue or ask a question about CodeQL + url: https://github.com/github/codeql/issues/new/choose + about: Please create issues and ask questions in the `github/codeql` repository. From 9de7d86bc52075f7070461c422a11a032b96699d Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 4 May 2023 00:22:45 -0400 Subject: [PATCH 095/196] Update changelog for 2.13.1 --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb2cfea..798de16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ you know what to do). --> +## Release 2.13.1 (2023-05-03) + +### Bugs fixed + +- Fixed a bug in `codeql database upload-results` where the subcommand + would fail with "A fatal error occurred: Invalid SARIF.", reporting + an `InvalidDefinitionException`. This issue occurred when the SARIF + file contained certain kinds of diagnostic information. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.7. + ## Release 2.13.0 (2023-04-20) ### Known issues From c884191d94b784749f80fae9e7af906c0d6213ec Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Wed, 31 May 2023 12:32:54 +0100 Subject: [PATCH 096/196] Update CHANGELOG.md for 2.13.3 --- CHANGELOG.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 798de16..17d0cf7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,40 @@ you know what to do). --> +## Release 2.13.3 (2023-05-31) + +### New features + +- This release enhances our preliminary Swift support, setting the stage for the upcoming public beta. + +- The `codeql database bundle` command now supports the `--[no]-include-temp` + option. When enabled, this option will include the `temp` folder of the + database directory in the zip file of the bundled database. This folder + includes generated packages and queries, and query suites. + +- The structured log produced by `codeql generate log-summary` now includes a Boolean `isCached` + field for predicate events, where a `true` value indicates the predicate is a wrapper implementing + the `cached` annotation on another predicate. The wrapper depends on the underlying predicate that + the annotation was found on, and will usually have the same name, but it has a separate `raHash`. + +### Bugs fixed + +- Fixed a bug that could cause the compiler to infer incorrect binding sets for + non-direct calls to overriding member predicates that have stronger binding sets + than their root definitions. + +- Fixed a bug that could have caused the compiler to incorrectly infer that a + class matched a type signature. The bug only affected classes with overriding + member predicates that had stronger binding sets than their root definitions. + +- Fixed a bug where a query could not be run from VS Code + when there were packs nested within sibling directories + of the query. + +## Release 2.13.2 + +This release was skipped. + ## Release 2.13.1 (2023-05-03) ### Bugs fixed From 97d73d019c6d837d5f193344d1512a3b000ec79e Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Mon, 19 Jun 2023 20:56:48 +0100 Subject: [PATCH 097/196] Update changelog for 2.13.4 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 17d0cf7..1658974 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ you know what to do). --> +## Release 2.13.4 (2023-06-19) + +### New features + +- Temporary files and folders created by the CodeQL CLI will now be cleaned up + when each CLI command (and its internal JVM) shuts down normally. + +### Bugs fixed + +- Fixed an issue where indirect build tracing did not work in Azure DevOps + pipeline jobs in Windows containers. To use indirect build tracing in such + environments, ensure both the `--begin-tracing` and + `--trace-process-name=CExecSvc.exe` arguments are passed to + `codeql database init`. +- Improved the error message for the `codeql pack create` command when the pack + being published has a dependency with no scope in its name. + ## Release 2.13.3 (2023-05-31) ### New features From 799e59b37b8e10e7625fca9757a32c11745785d9 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 13 Jul 2023 09:40:21 -0700 Subject: [PATCH 098/196] Update changelog for 2.13.5 and 2.14.0 --- CHANGELOG.md | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1658974..27743db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,88 @@ you know what to do). --> +## Release 2.14.0 (2023-07-13) + +### Potentially breaking changes + +- The legacy option `--search-path` will now be used, if provided, when + searching for the dependencies of packages that have no lock file. +- CodeQL query packs that specify their dependencies using the legacy + `libraryPathDependencies` property in `qlpack.yml`/`codeql-pack.yml` + files are no longer permitted to contain a `codeql-pack.lock.yml` lock file. + This will lead to a compilation error. This change is intended to prevent + confusing behavior arising from a mix of legacy (unversioned) and modern + (versioned) package dependencies. To fix this error, either delete the lock + file, or convert `libraryPathDependencies` to `dependencies`. +- CodeQL CLI commands that create packages or update package lock files, such + as `codeql pack publish` and `codeql pack create`, will no longer work on + query packs that specify their dependencies using the legacy + `libraryPathDependencies` property. To fix this error, convert + `libraryPathDependencies` to `dependencies`. + +### Deprecations + +- Missing override annotations on class member predicates now raise + errors rather than warnings. This is to avoid confusion with the + shadowing behaviour in the presence of final member predicates. + ```ql + class Foo extends Base { + final predicate foo() { ... } + + predicate bar() { ... } + } + + class Bar extends Foo { + // This method shadows Foo::foo. + predicate foo() { ... } + + // This used to override Foo::bar with a warning, now raises error. + predicate bar() { ... } + } + ``` + +### Improvements + +- Unqualified imports can now be marked as deprecated to indicate that the + import may be removed in the future. Usage of names only reachable through + deprecated imports will generate deprecation warnings. +- Classes declared inside a parameterized modules can final extend + parameters of the module as well as types that are declared outside + the parameterized module. +- Fields are fully functional when extending types from within a module + instantiation. +- Files with a `.yaml` extension will now be included in compiled + CodeQL packs. Previously, files with this extension were excluded + even though `.yml` files were included. +- When interpreting results (e.g., using `bqrs interpret` or + `database interpret-results`), extra placeholders in alert messages are + treated as normal text. Previously, results with more placeholders than + placeholder values were skipped. +- Windows users of the CodeQL extension for VS Code will see faster start times. +- In VS Code, errors in the current file are rechecked when dependencies change. +- In VS Code, autocomplete in large QL files is now faster. +- Member predicates can shadow final member predicates of the same arity even + when the signatures are not fully matching. + +### Bugs fixed + +- Fixed super calls on final base classes (or final aliases) so that they + are now dispatched the same way as super calls on instanceof supertypes. +- Fixed a bug where running `codeql database finalize` with a large number of + threads would fail due to running out of file descriptors. +- Fixed a bug where `codeql database create --overwrite` would not work + with database clusters. +- Fixed a bug where the CodeQL documentation coverage statistics were + incorrect. +- Fixed a bug where the generated CodeQL libarary documentation could + generate invalid uris on windows. + +## Release 2.13.5 (2023-07-05) + +### New Features + +- The Swift extractor now supports Swift 5.8.1. + ## Release 2.13.4 (2023-06-19) ### New features From b7f6a88344b9f30f2f1181905709e6c410ae87b4 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Thu, 27 Jul 2023 12:56:35 +0100 Subject: [PATCH 099/196] Update CHANGELOG.md for 2.14.1 release --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 27743db..fbe8c9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.14.1 (2023-07-27) + +- There are no user-facing changes in this release. + ## Release 2.14.0 (2023-07-13) ### Potentially breaking changes From 2e52d63073ce778686df941a0050bdf4f84ea8ac Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Fri, 11 Aug 2023 12:10:22 +0100 Subject: [PATCH 100/196] Update changelog for 2.14.2 --- CHANGELOG.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index fbe8c9b..718394e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,35 @@ you know what to do). --> +## Release 2.14.2 (2023-08-11) + +### Breaking changes + +- The functionality provided by the `codeql execute query-server` subcommand + has been removed. The subcommand now responds to all JSON RPC requests with + an error response. Correspondingly, this release is no longer compatible with + versions of the CodeQL extension for Visual Studio Code prior to 1.7.6. + + This change also breaks third-party CodeQL IDE integrations that still rely + on the `codeql execute query-server` subcommand. Maintainers of such CodeQL + IDE integrations should migrate to the `codeql execute query-server2` + subcommand at the earliest opportunity. + +### Improvements + +- Switched from prefix filtering of autocomplete suggestions in the language + server to client-side filtering. This improves autocomplete suggestions in + contexts with an autocompletion prefix. + +- The CodeQL language server now checks query metadata for errors. This allows + Visual Studio Code users to see errors in their query metadata without needing + to compile the query. + +### Bugs fixed + +- Fixed bug that made the `--warnings=hide` option do nothing in + `codeql database analyze` and other commands that _evaluate_ queries. + ## Release 2.14.1 (2023-07-27) - There are no user-facing changes in this release. From 14629faa668cf2fa4baae025e581e7448a29970f Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 25 Aug 2023 11:48:38 -0400 Subject: [PATCH 101/196] Update changelog for 2.14.3 --- CHANGELOG.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 718394e..351a35f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,35 @@ you know what to do). --> +## Release 2.14.3 (2023-08-25) + +### Breaking changes + +- The `.tool.extensions` property in the SARIF generated by `codeql database analyze` now contains the + following packs: + + - The containing query pack for each query that was evaluated. + - Each model pack that was specified via the `--model-packs` option, regardless of whether + that model pack affected any of the evaluated queries. + + Library packs are no longer included in the list. + + Previously, this property contained every query and library pack that was available on the search + path, regardless of whether that pack was used during the evaluation. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.8. + +- When `codeql test` generates `.actual` files, they will in some cases + list the query predicates in a different order than past versions. + There is no need to update `.expected` files, as `codeql test` sorts + their results accordingly before diffing. + However, when there are genuine changes in expected results, the + generated `.actual` file can show additional changes against the + `.expected` due to the reordering. + ## Release 2.14.2 (2023-08-11) ### Breaking changes From c414ee55767a368f24c77c349b753b0d4ae72522 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Tue, 12 Sep 2023 07:26:09 -0700 Subject: [PATCH 102/196] Update changelog for 2.14.4 --- CHANGELOG.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 351a35f..4488300 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,39 @@ you know what to do). --> +## Release 2.14.4 (2023-09-12) + +### Potentially breaking changes + +- The CodeQL CLI no longer supports the `SEMMLE_JAVA_ARGS` environment variable. + All previous versions of the CodeQL CLI perform command substitution on the + `SEMMLE_JAVA_ARGS` value (for example, replacing `'$(echo foo)'` with `'foo'`) + when starting a new Java virtual machine, which, depending on the execution + environment, may have security implications. Users are advised to check their + environments for possible `SEMMLE_JAVA_ARGS` misuse. + +### New Features + +- The Java extractor now supports files that use Lombok. + +### Bugs fixed + +- `codeql database init` (and `github/codeql-action/init@v2` on GitHub Actions) + should no longer hang or crash for traced languages on 64-bit Windows machines + when certain antivirus software is installed. +- During `codeql pack create` and `codeql pack publish`, a source version of a + pack coming from `--additional-packs` can explicitly be used to override a + requested pack version even if this source version is incompatible with the + requested version in the pack file. Previously, this would fail with a + confusing error message. +- Fixed a bug where `codeql database interpret-results` hangs when a path query + produces a result that has no paths from source to sink. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL + CLI has been updated to version 17.0.8. + ## Release 2.14.3 (2023-08-25) ### Breaking changes From 4dac984b095a131e349f572feebbff1d11ad49fa Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 14 Sep 2023 08:07:04 -0700 Subject: [PATCH 103/196] Update changelog for 2.14.5 --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4488300..2159fd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.14.5 (2023-09-14) + +### Bugs fixed + +- Fixed a JavaScript extractor crash that was introduced in 2.14.4. + ## Release 2.14.4 (2023-09-12) ### Potentially breaking changes From 426f50b9e3290ec1b1ee2fc5c8369663bd3a02ca Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 26 Sep 2023 14:05:52 +0100 Subject: [PATCH 104/196] Update CHANGELOG for 2.14.6 --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2159fd9..2ccb777 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.14.6 (2023-09-26) + +### Bugs fixed + +- The tracking of RAM usage has been improved. This fixes some cases + where CodeQL uses more RAM than requested. + ## Release 2.14.5 (2023-09-14) ### Bugs fixed From 63d0adb66656c7da08f24e183377c9c5b44330b5 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Wed, 11 Oct 2023 13:23:35 +0100 Subject: [PATCH 105/196] Update changelog for 2.15.0 --- CHANGELOG.md | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ccb777..200830b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,67 @@ you know what to do). --> +## Release 2.15.0 (2023-10-11) + +### Deprecations + +- `pragma[assume_small_delta]` is now deprecated. The pragma has no effect and + should be removed. + +- Missing override annotations on class fields now raise errors rather than + warnings. This is to avoid confusion with the shadowing behavior in the + presence of final fields. + +- The CodeQL CLI no longer supports ML-powered alerts. For more information, + including details of our work in the AI-powered security technology space, + see + "[CodeQL code scanning deprecates ML-powered alerts](https://github.blog/changelog/2023-09-29-codeql-code-scanning-deprecates-ml-powered-alerts/)." + +### New Features + +- The output of `codeql version --format json` now includes a `features` + property. Each key in the map identifies a feature of the CodeQL CLI. The + value for a key is always `true`. Going forward, whenever a significant new + feature is added to the CodeQL CLI, a corresponding entry will be added to the + `features` map. This is intended to make it easier for tools that invoke the + CodeQL CLI to know if the particular version of the CLI they are invoking + supports a given feature, without having to know exactly what CLI version + introduced that feature. + +### Improvements + +- You can now specify the CodeQL languages C/C++, Java/Kotlin, and + JavaScript/TypeScript using `--language c-cpp`, `--language java-kotlin`, and + `--language javascript-typescript` respectively. These new CodeQL language + names convey more clearly what languages each CodeQL language will analyze. + + You can also reference these CodeQL languages via their secondary language + names (C/C++ via `--language c` or `--language cpp`, Java/Kotlin via + `--language java` or `--language kotlin`, and JavaScript/TypeScript via + `--language javascript` or `--language typescript`), however we recommend you + refer to them via the new primary CodeQL language names for improved clarity. + +- CodeQL now respects custom home directories set by the `$HOME` environment + variable on MacOS and Linux and `%USERPROFILE%` on Windows. When set, CodeQL + will use the variable's value to change the default location of downloaded + packages and the global compilation cache. + +- This release improves the quality of + [file coverage information](https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/about-the-tool-status-page#using-the-tool-status-page) + for repositories that vendor their dependencies. This is currently supported + for Go and JavaScript projects. + +### Bugs fixed + +- Fixed an issue with analyzing Python projects using Python 3.12. + +### QL language improvements + +- The QL language now has two new methods `codePointAt` and `codePointCount` on + the `string` type. The methods both return integers and act the same as the + similarly named Java methods on strings. For example, `"abc".codePointAt(2)` + is `99` and `("a" + 128512.toUnicode() + "c").codePointAt(1)` is a `128512`. + ## Release 2.14.6 (2023-09-26) ### Bugs fixed From 631cf51d106114947d65b860b94576b2364e84af Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 19 Oct 2023 11:36:55 -0400 Subject: [PATCH 106/196] Changelog for 2.15.1 --- CHANGELOG.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 200830b..2a78b57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,40 @@ you know what to do). --> +## Release 2.15.1 (2023-10-19) + +### Potentially Breaking Changes + +- The query server's `evaluation/trimCache` command was previously equivalent to the `codeql database cleanup --mode=gentle` CLI command, but is now equivalent to using `--mode=normal`. The new meaning of the command is to clear the entire evaluation cache of a database except for predicates annotated with the `cached` keyword. + +### Deprecations + +- The accepted values of the `codeql database cleanup --mode=` command line option have been renamed to bring them in line with what they are called in the VSCode extension and the query server: + - `--mode=brutal` is now `--mode=clear`. + - `--mode=normal` is now `--mode=trim`. + - `--mode=light` is now `--mode=fit`. + - The old names are deprecated, but will be accepted for backwards-compatibility reasons until further notice. + +### New Features + +- The output of `codeql version --format json` now includes a `features` property. Each key in the + map identifies a feature of the CodeQL CLI. The value for a key is always `true`. Going forward, + whenever a significant new feature is added to the CodeQL CLI, a corresponding entry will be added to the + `features` map. This is intended to make it easier for tools that invoke the CodeQL CLI to know + if the particular version of the CLI they are invoking supports a given feature, without having to + know exactly what CLI version introduced that feature. + +### Improvements + +- The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. + +### Bugs fixed + +- Fixed a bug where the `$CODEQL_JAVA_HOME` environment variable was + erroneously ignored for certain subsidiary Java processes started by + `codeql`. +- Fixed a bug in the CodeQL build tracer on Apple Silicon machines that prevented database creation if System Integrity Protection was disabled. + ## Release 2.15.0 (2023-10-11) ### Deprecations From d75a9e63552d8eeda5840bbdba74365f994336c0 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Fri, 20 Oct 2023 13:52:11 +0100 Subject: [PATCH 107/196] Remove duplicate changelog entry --- CHANGELOG.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a78b57..7e4a79d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,15 +31,6 @@ - `--mode=light` is now `--mode=fit`. - The old names are deprecated, but will be accepted for backwards-compatibility reasons until further notice. -### New Features - -- The output of `codeql version --format json` now includes a `features` property. Each key in the - map identifies a feature of the CodeQL CLI. The value for a key is always `true`. Going forward, - whenever a significant new feature is added to the CodeQL CLI, a corresponding entry will be added to the - `features` map. This is intended to make it easier for tools that invoke the CodeQL CLI to know - if the particular version of the CLI they are invoking supports a given feature, without having to - know exactly what CLI version introduced that feature. - ### Improvements - The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. From e2179230b1147a4d5ecf7e5e4cdd20ca0c75393d Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Mon, 13 Nov 2023 12:17:56 +0000 Subject: [PATCH 108/196] Update Changelog for 2.15.2 release --- CHANGELOG.md | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e4a79d..5739b58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,46 @@ you know what to do). --> +## Release 2.15.2 (2023-11-13) + +### Breaking changes + +- C++ extraction has been updated to output more accurate C++ value categories. + This may cause unexpected alerts on databases extracted with an up-to-date CodeQL + when the queries are part of a query pack that was compiled with an earlier CodeQL. + To resolve this, please recompile the query pack with the latest CodeQL. + +### New features + +- `codeql database analyze` and `codeql database interpret-results` can now + output human-readable analysis summaries in a new format. This format provides file coverage + information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when the `GITHUB_SERVER_URL` and `GITHUB_REPOSITORY` environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server + version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. +- CodeQL now supports + distinguishing file coverage information between related languages C and C++, Java and Kotlin, + and JavaScript and TypeScript. By default, file coverage information for each + of these pairs of languages is grouped together. To enable specific file coverage information for these languages, pass the + `--sublanguage-file-coverage` flag when initializing the database (with `codeql database create` or `codeql database init`) and when analyzing the database (with `codeql database analyze` or `codeql database interpret-results`). If you are uploading results to a GitHub instance, this flag requires GitHub.com or GitHub Enterprise Server version 3.12 or later. +- All CLI commands now support `--common-caches`, which controls the location of the + cached data that is persisted between several runs of the CLI, such as downloaded QL packs + and compiled query plans. + +### Improvements + +- Model packs that are used in an analysis will now be included in an output SARIF results file. All model packs now include the `isCodeQLModelPack: true` property in their tool component property bag. +- The default formatting of DIL now more closely resembles equivalent QL code. + +### Bugs fixed + +- Fixed a bug where `codeql github upload-results` would report a 403 error when attempting to upload to a GitHub Enterprise Server instance. +- Fixed a bug in Python extraction where UTF-8 characters would cause + logging to fail on systems with non-UTF-8 default system encoding (for example, Windows systems). +- The `resolve qlpacks --kind extension` command no longer resolves + extensions packs from the search path. This matches the behavior of + `resolve extensions-by-pack` and will ensure that extensions which are + resolved by `resolve qlpacks --kind extension` can also be resolved by + `resolve extensions-by-pack`. + ## Release 2.15.1 (2023-10-19) ### Potentially Breaking Changes @@ -25,7 +65,7 @@ ### Deprecations -- The accepted values of the `codeql database cleanup --mode=` command line option have been renamed to bring them in line with what they are called in the VSCode extension and the query server: +- The accepted values of the `--mode` option for `codeql database cleanup` have been renamed to bring them in line with what they are called in the VSCode extension and the query server: - `--mode=brutal` is now `--mode=clear`. - `--mode=normal` is now `--mode=trim`. - `--mode=light` is now `--mode=fit`. @@ -34,6 +74,9 @@ ### Improvements - The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. +- The syntax of DIL now more closely resembles the QL source code that it is + compiled from. In particular, conjunctions and disjunctions now use the + familiar `and` and `or` keywords, and clauses are enclosed in curly braces. ### Bugs fixed From 6f34712b475e5c949c6cb08d88835b9db169bf09 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Mon, 13 Nov 2023 12:20:34 +0000 Subject: [PATCH 109/196] Revert 2.15.1 changes I assume these were alterations made after the release prep workflows were run. --- CHANGELOG.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5739b58..2489a4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,7 +65,7 @@ ### Deprecations -- The accepted values of the `--mode` option for `codeql database cleanup` have been renamed to bring them in line with what they are called in the VSCode extension and the query server: +- The accepted values of the `codeql database cleanup --mode=` command line option have been renamed to bring them in line with what they are called in the VSCode extension and the query server: - `--mode=brutal` is now `--mode=clear`. - `--mode=normal` is now `--mode=trim`. - `--mode=light` is now `--mode=fit`. @@ -74,9 +74,6 @@ ### Improvements - The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. -- The syntax of DIL now more closely resembles the QL source code that it is - compiled from. In particular, conjunctions and disjunctions now use the - familiar `and` and `or` keywords, and clauses are enclosed in curly braces. ### Bugs fixed From 3ac55ee10a73eea5ba45066f861392d618792eeb Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Wed, 22 Nov 2023 10:43:01 +0100 Subject: [PATCH 110/196] Update CHANGELOG.md --- CHANGELOG.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2489a4c..059feae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,54 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.15.3 (2023-11-22) + +### New features + +- A new compilation flag (`--fail-on-ambiguous-relation-name`) has been added to specify + that compilation should fail if the compiler generates an ambiguous relation name. +- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the + linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of + database completeness) for C++ projects where this part of database creation consumes + too much memory. This option is available in the commands `database create`, + `database finalize`, `database import`, `dataset import`, `test extract`, and + `test run`. +- The CodeQL language server now provides basic support for Rename, and you can + now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The + current Rename support is less a refactoring tool and more a labor-saving + device. You may have to perform some manual edits after using Rename, but it + should still be faster and less work than renaming a symbol manually. +- `codeql database analyze` now defaults to include markdown query help for all custom + queries with help files available. To change the default behaviour you can pass the + new flag `--sarif-include-query-help`, which provides the options `always` (which + includes query help for all queries), `custom_queries_only` (the default) and `never` + (which does not include query help for any query). The existing flag + `--sarif-add-query-help` has been deprecated and will be removed in a future release. + +### Improvements + +- The Find References feature in the CodeQL language server now supports all + CodeQL identifiers and offers improved performance compared to CodeQL CLI + 2.14 releases. +- The compiler generates shorter human-readable DIL and RA relation names. Due + to use of an extended character set, full VS Code support for short relation + names requires VS Code extension 1.9.4 or newer. +- `codeql database create` and `codeql database finalize` now log more diagnostic + information during database finalization, including the size of each relation, their + total size, and the rate at which they were written to disk. + +### Bugs fixed + +- Fixed an internal error in the compiler when arguments to the `codePointCount` string + primitive were not bound. +- Fixed a bug where `codeql database finalize` would fail if a + database under construction was moved between machines between + `codeql database init` and `codeql database finalize`. This should + now work, as long as both commands are run by the same _release_ of + the CodeQL CLI and the extractors used are the ones bundled with the + CLI. +- Fixed a bug where `codeql database run-queries` would fail in some + circumstances when the database path included an `@`. ## Release 2.15.2 (2023-11-13) @@ -65,7 +113,7 @@ ### Deprecations -- The accepted values of the `codeql database cleanup --mode=` command line option have been renamed to bring them in line with what they are called in the VSCode extension and the query server: +- The accepted values of the `--mode` option for `codeql database cleanup` have been renamed to bring them in line with what they are called in the VSCode extension and the query server: - `--mode=brutal` is now `--mode=clear`. - `--mode=normal` is now `--mode=trim`. - `--mode=light` is now `--mode=fit`. @@ -74,6 +122,9 @@ ### Improvements - The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. +- The syntax of DIL now more closely resembles the QL source code that it is + compiled from. In particular, conjunctions and disjunctions now use the + familiar `and` and `or` keywords, and clauses are enclosed in curly braces. ### Bugs fixed From 21314e357e71c63c41f1a5e753bd6d95717fdbeb Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 11 Dec 2023 08:42:46 -0800 Subject: [PATCH 111/196] Revise changelog for 2.15.3 --- CHANGELOG.md | 49 ++++++++++++++++++++++--------------------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 059feae..d727e50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,34 +20,31 @@ ### New features -- A new compilation flag (`--fail-on-ambiguous-relation-name`) has been added to specify - that compilation should fail if the compiler generates an ambiguous relation name. -- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the - linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of - database completeness) for C++ projects where this part of database creation consumes - too much memory. This option is available in the commands `database create`, - `database finalize`, `database import`, `dataset import`, `test extract`, and - `test run`. -- The CodeQL language server now provides basic support for Rename, and you can - now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The - current Rename support is less a refactoring tool and more a labor-saving - device. You may have to perform some manual edits after using Rename, but it - should still be faster and less work than renaming a symbol manually. - `codeql database analyze` now defaults to include markdown query help for all custom queries with help files available. To change the default behaviour you can pass the new flag `--sarif-include-query-help`, which provides the options `always` (which includes query help for all queries), `custom_queries_only` (the default) and `never` (which does not include query help for any query). The existing flag `--sarif-add-query-help` has been deprecated and will be removed in a future release. +- The new (advanced) command-line option `--[no-]linkage-aware-import` disables the + linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of + database completeness) for C++ projects where this part of database creation consumes + too much memory. This option is available in the commands `database create`, + `database finalize`, `database import`, `dataset import`, `test extract`, and + `test run`. +- The CodeQL language server now provides basic support for Rename, and you can now use + the Rename Symbol functionality in Visual Studio Code for CodeQL. The current Rename + support is less a refactoring tool and more a labor-saving device. You may have to + perform some manual edits after using Rename, but it should still be faster and less + work than renaming a symbol manually. ### Improvements -- The Find References feature in the CodeQL language server now supports all - CodeQL identifiers and offers improved performance compared to CodeQL CLI - 2.14 releases. -- The compiler generates shorter human-readable DIL and RA relation names. Due - to use of an extended character set, full VS Code support for short relation - names requires VS Code extension 1.9.4 or newer. +- The Find References feature in the CodeQL language server now supports all CodeQL + identifiers and offers improved performance compared to CodeQL CLI 2.14 releases. +- The compiler generates shorter human-readable DIL and RA relation names. Due to use + of an extended character set, full VS Code support for short relation names requires + VS Code extension 1.9.4 or newer. - `codeql database create` and `codeql database finalize` now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk. @@ -56,14 +53,12 @@ - Fixed an internal error in the compiler when arguments to the `codePointCount` string primitive were not bound. -- Fixed a bug where `codeql database finalize` would fail if a - database under construction was moved between machines between - `codeql database init` and `codeql database finalize`. This should - now work, as long as both commands are run by the same _release_ of - the CodeQL CLI and the extractors used are the ones bundled with the - CLI. -- Fixed a bug where `codeql database run-queries` would fail in some - circumstances when the database path included an `@`. +- Fixed a bug where `codeql database finalize` would fail if a database under construction + was moved between machines between `codeql database init` and `codeql database finalize`. + This should now work, as long as both commands are run by the same _release_ of the + CodeQL CLI and the extractors used are the ones bundled with the CLI. +- Fixed a bug where `codeql database run-queries` would fail in some circumstances when + the database path included an `@`. ## Release 2.15.2 (2023-11-13) From 51e566086be89a90231de896d9ed5d3b2f78566c Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 11 Dec 2023 08:43:38 -0800 Subject: [PATCH 112/196] Update changelog for 2.15.4 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d727e50..9e9af2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,14 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.15.4 (2023-12-11) + +### Improvements + +- Parallelism in the evaluator has been improved, resulting in faster analysis when + running with many threads, particularly for large databases. + ## Release 2.15.3 (2023-11-22) ### New features From e67d1c08b78afc431211f5907e83d366691d6337 Mon Sep 17 00:00:00 2001 From: Aditya Sharad <6874315+adityasharad@users.noreply.github.com> Date: Wed, 20 Dec 2023 08:57:23 -0800 Subject: [PATCH 113/196] Update changelog for 2.15.5 --- CHANGELOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e9af2a..7c1e451 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,29 @@ you know what to do). --> +## Release 2.15.5 (2023-12-20) + +### New features + +- A new extractor option has been added to the JavaScript/TypeScript extractor. + Set the environment variable `CODEQL_EXTRACTOR_JAVASCRIPT_OPTION_SKIP_TYPES` + to `true` to skip the extraction of types in TypeScript files. + Use this to speed up extraction if your codebase has a high volume of + TypeScript type information that causes a noticeable bottleneck for + TypeScript extraction. The majority of analysis results should be preserved + even when no types are extracted. + +### Bugs fixed + +- Fixed an issue where CodeQL would sometimes incorrectly report that no files + were scanned when running on Windows. + This affected the human-readable summary produced by `codeql database analyze` + and `codeql database interpret-results`, but did not impact the file coverage + information produced in the SARIF output and displayed on the tool status page. +- When analyzing Swift codebases, CodeQL build tracing will now ignore the + `codesign` tool. This prevents errors in build commands or workflows on macOS + that include both CodeQL and code signing. + ## Release 2.15.4 (2023-12-11) ### Improvements From 1d24865e275defb303f35f4be496e780fbf48a5c Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Fri, 5 Jan 2024 14:23:59 +0000 Subject: [PATCH 114/196] Add change note re Java 21 support in 2.15.4 --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c1e451..2ed17bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,11 @@ ## Release 2.15.4 (2023-12-11) +### New features + +- Java 21 is now fully supported, including support for new language features such as + pattern switches and record patterns. + ### Improvements - Parallelism in the evaluator has been improved, resulting in faster analysis when From fb86fb9d41b02ee0c7de1dda971ffbc1a0a3e442 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 16 Jan 2024 13:00:26 +0000 Subject: [PATCH 115/196] Update CHANGELOG.md for 2.16.0 --- CHANGELOG.md | 51 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ed17bc..1290688 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,52 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.16.0 (2024-01-16) + +### New Features + +- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. + +### Improvements + +- The `Experimental` flag has been removed from all packaging and related commands. +- The RA pretty-printer omits names of internal RA nodes and pretty-prints + binary unions with nested internal unions as n-ary unions. VS Code extension + v1.11.0 or newer is required to compute join order badness metrics in VS Code + for the new RA format. + + +### Potentially breaking changes + +- The Python extractor will no longer extract dependencies by default. See https://github.blog/changelog/2023-07-12-code-scanning-with-codeql-no-longer-installs-python-dependencies-automatically-for-new-users/ for more context. In versions until 2.17.0, it will be possible to restore the old behavior by setting `CODEQL_EXTRACTOR_PYTHON_FORCE_ENABLE_LIBRARY_EXTRACTION_UNTIL_2_17_0=1`. +- The `--ram` option to `codeql database run-queries` and other + commands that execute queries is now interpreted more strictly. + Previously it was mostly a rough hint for how much memory to use, + and the actual memory footprint of the CodeQL process could be + hundreds of megabytes higher. From this release, CodeQL tries harder + to keep its _total_ memory consumption during evaluation below the + given limit. + + The new behavior yields more predictable memory use, but since it + works by allocating less RAM, it can lead to more use of _disk_ + storage for intermediate results compared to earlier releases with + the same `--ram` value, and consequently a slight performance + loss. In rare cases, for large databases, analysis may fail with a + Java `OutOfMemoryError`. + + The cure for this is to increase `--ram` to be closer to the amount + of memory actually available for CodeQL. As a rule of thumb, it will + usually be possible to increase the value of `--ram` by 700 MB or + more, without actually using more resources than release 2.15.x + would with the old setting. An exact amount cannot stated, however, + since the actual memory footprint in earlier releases depended on + factors such as the size of the databases that were not fully taken + into account. + + If you use the CodeQL Action, you do not need to do anything unless + you have manually overridden the Action's RAM setting. The Action + will automatically select a `--ram` setting that matches the version + of the CLI it uses. ## Release 2.15.5 (2023-12-20) @@ -33,7 +79,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the @@ -44,8 +90,7 @@ ### New features -- Java 21 is now fully supported, including support for new language features such as - pattern switches and record patterns. +- Java 21 is now fully supported, including support for new language features such as pattern switches and record patterns. ### Improvements From 03ea17bd85e4e9f8d0a0ee16be392a1a055344a7 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Thu, 25 Jan 2024 13:57:50 +0000 Subject: [PATCH 116/196] Update changelog for 2.16.1 --- CHANGELOG.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1290688..4531ba6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,22 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.16.1 (2024-01-25) + +### Improvements + +- When executing the `codeql database init` command, the CodeQL runner + executable path is now stored in the `CODEQL_RUNNER` environment variable. + Users of indirect tracing on MacOS with System Integrity Protection enabled + who previously had trouble with indirect tracing should prefix their build + command with this path. For example, `$CODEQL_RUNNER build.sh`. + +### QL language improvements + +- Name clashes between weak aliases (i.e. aliases that are not final aliases of + non-final entities) of the same target no longer cause ambiguity errors. + ## Release 2.16.0 (2024-01-16) ### New Features From 0a621d68fb21b5a9bdfc90b4313da47df74b6b00 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Mon, 12 Feb 2024 13:17:10 -0500 Subject: [PATCH 117/196] Update chanelog for 2.16.2 --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4531ba6..7eac2fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.16.2 (2024-02-12) + +- There are no user-facing changes in this release. + ## Release 2.16.1 (2024-01-25) ### Improvements From f6b18e6381860d453f6d3bff8de4f99eeb0a218b Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 22 Feb 2024 11:39:00 +0000 Subject: [PATCH 118/196] Update CHANGELOG.md for release 2.16.3 --- CHANGELOG.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7eac2fb..6042687 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,32 @@ you know what to do). --> +## Release 2.16.3 (2024-02-22) + +### Security patches + +- Fixes CVE-2024-25129, a limited data exfiltration vulnerability that + could be triggered by untrusted databases or QL packs. See the + [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gf8p-v3g3-3wph) + for more information. + +### New Features + +- A new extractor option has been added to the Python extractor. + Set the new extractor option `python_executable_name` or the environment variable + `CODEQL_EXTRACTOR_PYTHON_OPTION_PYTHON_EXECUTABLE_NAME` to one of `py`, `python` or `python3` + to override the default Python executable search and selection behavior of the Python extractor. + For example, on Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. Setting this extractor option or environment variable allows + overriding this behavior to look for a different name like `python` or `python3`. + More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + +### Bugs fixed + +- Fixed a bug where CodeQL may produce an invalid database when it exhausts + all available ID numbers. Now it detects the condition and reports an + error instead. + ## Release 2.16.2 (2024-02-12) - There are no user-facing changes in this release. @@ -99,7 +125,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From d6584c92c977a8553f4368fb38479a69335159b3 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 22 Feb 2024 11:43:27 +0000 Subject: [PATCH 119/196] Replace nbsp with regular space --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6042687..7e3c5f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -125,7 +125,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 6ffe4091e50dc73fa58bdcbc1837de13131702e8 Mon Sep 17 00:00:00 2001 From: Henning Makholm Date: Thu, 22 Feb 2024 14:20:21 +0100 Subject: [PATCH 120/196] edit 2.16.3 changelog to explain how to set the extractor option --- CHANGELOG.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e3c5f7..490ae55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,12 +29,17 @@ ### New Features - A new extractor option has been added to the Python extractor. - Set the new extractor option `python_executable_name` or the environment variable - `CODEQL_EXTRACTOR_PYTHON_OPTION_PYTHON_EXECUTABLE_NAME` to one of `py`, `python` or `python3` - to override the default Python executable search and selection behavior of the Python extractor. - For example, on Windows machines, the Python extractor will expect to find `py.exe` on the - system `PATH` by default. Setting this extractor option or environment variable allows - overriding this behavior to look for a different name like `python` or `python3`. + Pass one of `--extractor-option python_executable_name=py` + or `--extractor-option python_executable_name=python` + or `--extractor-option python_executable_name=python3` + to `codeql database create` (or `codeql database trace-command` or, + for indirect tracing, `codeql database init`) to override the default + Python executable search and selection behavior of the Python + extractor. For example, on Windows machines, the Python extractor + will expect to find `py.exe` on the system `PATH` by default. + Setting this extractor option or environment variable allows + overriding this behavior to look for a different name. + More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed @@ -125,7 +130,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 8b68ee28b278049d2233fe756cb8f934830f0add Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Thu, 29 Feb 2024 13:54:49 +0000 Subject: [PATCH 121/196] update CLI changelog to match unified changelog --- CHANGELOG.md | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 490ae55..aa39eed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,19 +28,14 @@ ### New Features -- A new extractor option has been added to the Python extractor. - Pass one of `--extractor-option python_executable_name=py` - or `--extractor-option python_executable_name=python` - or `--extractor-option python_executable_name=python3` - to `codeql database create` (or `codeql database trace-command` or, - for indirect tracing, `codeql database init`) to override the default - Python executable search and selection behavior of the Python - extractor. For example, on Windows machines, the Python extractor - will expect to find `py.exe` on the system `PATH` by default. - Setting this extractor option or environment variable allows - overriding this behavior to look for a different name. - - More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). +- A new extractor option has been added to the Python extractor: `python_executable_name`. + You can use this option to override the default process the extractor uses to find and select a Python executable. + Pass one of `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. + If the Python executable has a different name, you can set the new extractor option to override this value and look for `python.exe` or `python3.exe`. + + For more information about using the extractor option with the CodeQL CLI, see [Extractor options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed From 68c904ca588f6cc12dd774bbafa25497af36dd8b Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Fri, 1 Mar 2024 11:39:10 +0000 Subject: [PATCH 122/196] word wrapped at 80 chars --- CHANGELOG.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa39eed..3d13adb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,14 +28,22 @@ ### New Features -- A new extractor option has been added to the Python extractor: `python_executable_name`. - You can use this option to override the default process the extractor uses to find and select a Python executable. - Pass one of `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for example: `codeql database create`. - - On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. - If the Python executable has a different name, you can set the new extractor option to override this value and look for `python.exe` or `python3.exe`. - - For more information about using the extractor option with the CodeQL CLI, see [Extractor options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). +- A new extractor option has been added to the Python extractor: + `python_executable_name`. You can use this option to override the default + process the extractor uses to find and select a Python executable. Pass one of + `--extractor-option python_executable_name=py` or `--extractor-option + python_executable_name=python` or `--extractor-option + python_executable_name=python3` to commands that run the extractor, for + example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. If the Python executable has a different name, you + can set the new extractor option to override this value and look for + `python.exe` or `python3.exe`. + + For more information about using the extractor option with the CodeQL CLI, see + [Extractor + options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed From 29581c71881c943e103e1590ad04f94cb131bfdc Mon Sep 17 00:00:00 2001 From: Angela P Wen Date: Mon, 11 Mar 2024 10:41:31 -0700 Subject: [PATCH 123/196] Update CHANGELOG.md for release 2.16.4 --- CHANGELOG.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d13adb..13070ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,39 @@ you know what to do). --> +## Release 2.16.4 (2024-03-11) + +### Potentially breaking changes + +- A number of internal command line options (`--builtin_functions_file`, `--clang_builtin_functions`, + `--disable-objc-default-synthesize-properties`, `--list_builtin_functions`, `--memory-limit-bytes`, + `--mimic_config`, and `--objc`) has been removed from the C/C++ extractor. It has never been + possible to pass these options through the CLI itself, but some customers with advanced setups may + have been passing them through internal undocumented interfaces. All of the removed options were + already no-ops, and will now generate errors. + + The `--verbosity` command line option has also been removed. The option was an alias for + `--codeql-verbosity`, which should be used instead. + +### Improvements + +- The frontend of the C/C++ extractor has been updated, improving the + extractor's reliability and increasing its ability to extract source code. + +### Bugs fixed + +- When parsing user-authored YAML files such as `codeql-pack.yml`, + `qlpack.yml`, `codeql-workspace.yml`, and any YAML file defining a data + extension, unquoted string values starting with a `*` character are now + correctly interpreted as YAML aliases. Previously, they were interpreted + as strings, but with the first character skipped. + + If you see a parse error similar to `while scanning an alias... unexpected` + `character found *(42)`,it likely means that you need to add quotes around + the indicated string value. The most common cause is unquoted glob patterns + that start with `*`, such as `include: **/*.yml`, which will need to be + quoted as `include: "**/*.yml"`. + ## Release 2.16.3 (2024-02-22) ### Security patches From 77e4d7fdcae8722662c94951f81886c4612e0491 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 21 Mar 2024 13:15:30 +0100 Subject: [PATCH 124/196] Update CHANGELOG.md for release 2.16.5 --- CHANGELOG.md | 54 ++++++++++++++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13070ae..41532b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.16.5 (2024-03-21) + +### New features + +- Beta support has been added for analyzing Java codebases without needing a working build. To enable + this, pass the `--build-mode none` option to `codeql database create`. + ## Release 2.16.4 (2024-03-11) ### Potentially breaking changes @@ -61,22 +68,19 @@ ### New Features -- A new extractor option has been added to the Python extractor: - `python_executable_name`. You can use this option to override the default - process the extractor uses to find and select a Python executable. Pass one of - `--extractor-option python_executable_name=py` or `--extractor-option - python_executable_name=python` or `--extractor-option - python_executable_name=python3` to commands that run the extractor, for - example: `codeql database create`. - - On Windows machines, the Python extractor will expect to find `py.exe` on the - system `PATH` by default. If the Python executable has a different name, you - can set the new extractor option to override this value and look for - `python.exe` or `python3.exe`. +- A new extractor option has been added to the Python extractor. + Pass one of `--extractor-option python_executable_name=py` + or `--extractor-option python_executable_name=python` + or `--extractor-option python_executable_name=python3` + to `codeql database create` (or `codeql database trace-command` or, + for indirect tracing, `codeql database init`) to override the default + Python executable search and selection behavior of the Python + extractor. For example, on Windows machines, the Python extractor + will expect to find `py.exe` on the system `PATH` by default. + Setting this extractor option or environment variable allows + overriding this behavior to look for a different name. - For more information about using the extractor option with the CodeQL CLI, see - [Extractor - options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed @@ -107,7 +111,7 @@ ### New Features -- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. +- Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. ### Improvements @@ -115,7 +119,7 @@ - The RA pretty-printer omits names of internal RA nodes and pretty-prints binary unions with nested internal unions as n-ary unions. VS Code extension v1.11.0 or newer is required to compute join order badness metrics in VS Code - for the new RA format. + for the new RA format. ### Potentially breaking changes @@ -212,7 +216,7 @@ identifiers and offers improved performance compared to CodeQL CLI 2.14 releases. - The compiler generates shorter human-readable DIL and RA relation names. Due to use of an extended character set, full VS Code support for short relation names requires - VS Code extension 1.9.4 or newer. + VS Code extension 1.9.4 or newer. - `codeql database create` and `codeql database finalize` now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk. @@ -242,7 +246,7 @@ - `codeql database analyze` and `codeql database interpret-results` can now output human-readable analysis summaries in a new format. This format provides file coverage information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when the `GITHUB_SERVER_URL` and `GITHUB_REPOSITORY` environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server - version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. + version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. - CodeQL now supports distinguishing file coverage information between related languages C and C++, Java and Kotlin, and JavaScript and TypeScript. By default, file coverage information for each @@ -591,7 +595,7 @@ member predicates that had stronger binding sets than their root definitions. - Fixed a bug where a query could not be run from VS Code - when there were packs nested within sibling directories + when there were packs nested within sibling directories of the query. ## Release 2.13.2 @@ -617,7 +621,7 @@ This release was skipped. ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI - system do not upgrade to this release, due to an issue with `codeql + system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. For more information, see the "Known issues" section for CodeQL 2.12.6. @@ -693,7 +697,7 @@ This release was skipped. ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI - system do not upgrade to this release, due to an issue with `codeql + system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. @@ -701,7 +705,7 @@ This release was skipped. causes the subcommand to fail with "A fatal error occurred: Invalid SARIF.", reporting an `InvalidDefinitionException`. - Customers who wish to use CodeQL 2.12.6 or 2.13.0 can + Customers who wish to use CodeQL 2.12.6 or 2.13.0 can work around the problem by passing `--no-sarif-include-diagnostics` to any invocations of `codeql database analyze` or `codeql database interpret-results`. @@ -867,8 +871,8 @@ This release was skipped. `codeql database create` now accounts for [`paths` and `paths-ignore` configuration](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#specifying-directories-to-scan). - In the VS Code extension, recursive calls will be marked with inlay - hints. These can be disabled with the global inlay hints setting - (`editor.inlayHints.enabled`). If you just want to disable them for + hints. These can be disabled with the global inlay hints setting + (`editor.inlayHints.enabled`). If you just want to disable them for codeql the settings can be scoped to just codeql files (language id is `ql`). See [Language Specific Editor Settings](https://code.visualstudio.com/docs/getstarted/settings#_language-specific-editor-settings) in the VS Code documentation for more information. From d9c8ca50e6f1186210bd2764f1f2b11e20e7f79d Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 26 Mar 2024 12:19:01 +0000 Subject: [PATCH 125/196] Update changelog for 2.16.6 --- CHANGELOG.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 41532b7..ff062a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.16.6 (2024-03-26) + +### Bugs fixed + +- Fixes a bug where extractor logs would be output at a lower than expected + verbosity level when using the `codeql database create` command. + ## Release 2.16.5 (2024-03-21) ### New features @@ -170,7 +177,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 7434aae9f5e2a2de1212e532fb6e993332476e8b Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 4 Apr 2024 08:26:56 -0700 Subject: [PATCH 126/196] Update CHANGELOG.md for release 2.17.0 --- CHANGELOG.md | 57 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 44 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ff062a5..d8a6f86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,34 @@ you know what to do). --> +## Release 2.17.0 (2024-04-04) + +### Deprecations + +- The `--[no-]analysis-summary-v2` and `--[no-]new-analysis-summary` options + that were used to enable (or disable) improved summary information printed at + the end of a `codeql database analyze` invocation are no longer supported. + Improved summary information is now enabled for all invocations. +- Support for overwriting default CodeQL SARIF run properties using the + `--sarif-run-property` command line option has been removed. This removes the + ability to overwrite the `semmle.formatSpecifier`, `metricResults`, and + `codeqlConfigSummary` properties in the SARIF run file. + +### Improvements + +- TRAP import (a part of `codeql database create` and `codeql database finalize`) + now performs better in low-memory situations. (Put another way, it now needs + less RAM to achieve the same performance as before.) + +- The worst-case performance of transitive closure computation (using + the `+` or `*` postfix operators or the `fastTC` higher-order + primitive in QL) has been greatly improved. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.2. + ## Release 2.16.6 (2024-03-26) ### Bugs fixed @@ -75,19 +103,22 @@ ### New Features -- A new extractor option has been added to the Python extractor. - Pass one of `--extractor-option python_executable_name=py` - or `--extractor-option python_executable_name=python` - or `--extractor-option python_executable_name=python3` - to `codeql database create` (or `codeql database trace-command` or, - for indirect tracing, `codeql database init`) to override the default - Python executable search and selection behavior of the Python - extractor. For example, on Windows machines, the Python extractor - will expect to find `py.exe` on the system `PATH` by default. - Setting this extractor option or environment variable allows - overriding this behavior to look for a different name. +- A new extractor option has been added to the Python extractor: + `python_executable_name`. You can use this option to override the default + process the extractor uses to find and select a Python executable. Pass one of + `--extractor-option python_executable_name=py` or `--extractor-option + python_executable_name=python` or `--extractor-option + python_executable_name=python3` to commands that run the extractor, for + example: `codeql database create`. + + On Windows machines, the Python extractor will expect to find `py.exe` on the + system `PATH` by default. If the Python executable has a different name, you + can set the new extractor option to override this value and look for + `python.exe` or `python3.exe`. - More detail can be found in [the extractor option documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). + For more information about using the extractor option with the CodeQL CLI, see + [Extractor + options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed @@ -177,7 +208,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 9a440a9881f4f831d4a60fae9b7cdaa300f71dcb Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Wed, 24 Apr 2024 16:35:27 +0100 Subject: [PATCH 127/196] Update CHANGELOG.md --- CHANGELOG.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d8a6f86..3ab0aac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,26 @@ you know what to do). --> +## Release 2.17.1 (2024-04-24) + +### Deprecations + +- The `--mode` option and `-m` alias to `codeql database create`, + `codeql database cleanup`, and `codeql dataset cleanup` has been + deprecated. Instead, use the new `--cache-cleanup` option, which has + identical behavior. + +### Improvements + +- Improved the diagnostic message produced when no code is processed + when creating a database. If a build mode was specified using + `--build-mode`, the message is now tailored to your build mode. + +### Miscellaneous + +- The `scc` tool used by the CodeQL CLI to calculate source code baseline + information has been updated to version [3.2.0](https://github.com/boyter/scc/releases/tag/v3.2.0). + ## Release 2.17.0 (2024-04-04) ### Deprecations From 6f3cb28e2eab62434cd376944bcfadf70e1f4371 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 7 May 2024 12:16:02 +0100 Subject: [PATCH 128/196] Update changelog for 2.17.2 --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ab0aac..df9ef16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,34 @@ you know what to do). --> +## Release 2.17.2 (2024-05-07) + +### Known issues + +- The beta support for analyzing Swift in this release and all + previous releases requires `g++-13` when running on Linux. Users + analyzing Swift using the `ubuntu-latest`, `ubuntu-22.04`, or + `ubuntu-20.04` runner images for GitHub Actions should update their + workflows to install `g++-13`. For more information, see [the runner + images + announcement](https://github.com/actions/runner-images/issues/9679). + +### Improvements + +- When uploading a SARIF file to GitHub using `codeql github + upload-results`, the CodeQL CLI now waits for the file to be + processed by GitHub. If any errors occurred during processing of the + analysis results, the command will log these and return a non-zero + exit code. To disable this behaviour, pass the + `--no-wait-for-processing` flag. + + By default, the command will wait for the SARIF file to be processed + for a maximum of 2 minutes, however this is configurable with the + `--wait-for-processing-timeout` option. +- The build tracer is no longer enabled when using the [`none` build + mode](https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#codeql-build-modes) + to analyze a compiled language, thus improving performance. + ## Release 2.17.1 (2024-04-24) ### Deprecations From 0b00cc963e1fbe20fdc4c9dbea1661f497fdb97a Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 17 May 2024 12:57:07 -0400 Subject: [PATCH 129/196] Update CHANGELOG.md --- CHANGELOG.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df9ef16..3a19eb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,21 @@ you know what to do). --> +## Release 2.17.3 (2024-05-17) + +### Improvements + +- The language server that our IDE integration is built on now defaults + to fine-grained dependency tracking for incremental error-checking + after file changes. This slightly improves the latency of refreshing + errors after local source code edits and will enable significant + speedups in the future. +- We now properly handle globs (such as `folder/**/*.py`) in `paths` configuration + to specify what files to include for Python analysis (see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#specifying-directories-to-scan). +- TRAP import (a part of `codeql database create` and `codeql database finalize`) + now supports allocating 2^32 IDs during the import process. The previous limit + was 2^31 IDs. + ## Release 2.17.2 (2024-05-07) ### Known issues @@ -256,7 +271,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 5bdc1ca4dde0776cbc72994f0158349628dfc2b3 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 17 May 2024 12:58:29 -0400 Subject: [PATCH 130/196] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a19eb6..f549eb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -271,7 +271,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From df51e673119ee0578d9cb4b2c01661f0c7c5e2ff Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Mon, 3 Jun 2024 09:01:05 -0700 Subject: [PATCH 131/196] Update CHANGELOG.md for release 2.17.4 --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f549eb2..29c516c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.17.4 (2024-06-03) + +- There are no user-facing changes in this release. + ## Release 2.17.3 (2024-05-17) ### Improvements From a10394a868747eba71cc617087f600f3878e03f5 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Wed, 12 Jun 2024 17:58:19 +0100 Subject: [PATCH 132/196] Update CHANGELOG.md for 2.17.5 --- CHANGELOG.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29c516c..6657c0d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,63 @@ you know what to do). --> +## Release 2.17.5 (2024-06-12) + +### Breaking changes + +- All the commands that output SARIF will output a minified version to reduce the size. + The `codeql database analyze`, `codeql database interpret-results`, `codeql generate query-help`, and `codeql bqrs interpret` commands support the option `--no-sarif-minify` to output a pretty printed SARIF file. + +- A number of breaking changes have been made to the `semmle-extractor-options` + functionality available for C and C++ CodeQL tests. + + - The Arm, Intel, and CodeWarrior compilers are no longer supported and the + `--armcc`, `--intel`, `--codewarrior` flags are now ignored, as are all the + flags that only applied to those compilers. + - The `--threads` and `-main-file-name` options, which did not have any effect + on tests, are now ignored. Any specification of these options as part of + `semmle-extractor-options` should be removed. + - Support for `--linker`, all flags that would only invoke the preprocessor, + and the `/clr` flag have been removed, as those flags would never produce any + usable test output. + - Support for the `--include_path_environment` flag has been removed. All include + paths should directly be specified as part of `semmle-extractor-options`. + - Microsoft C/C++ compiler response files specified via `@some_file_name` are + now ignored. Instead, all options should directly be specified as part of + `semmle-extractor-options`. + - Support for Microsoft `#import` preprocessor directive has been removed, as + support depends on the availability of the Microsoft C/C++ compiler, and + availability cannot be guaranteed on all platforms while executing tests. + - Support for the Microsoft `/EHa`, `/EHs`, `/GX`, `/GZ`, `/Tc`, `/Tp`, and `/Zl` + flags, and all `/RTC` flags have been removed. Any specification of these + options as part of `semmle-extractor-options` should be removed. + - Support for the Apple-specific `-F` and `-iframework` flags has been removed. + The `-F` flag can still be used by replacing `-F ` by + `--edg -F --edg `. Any occurrence of `-iframework ` should be + replaced by `--edg --sys_framework --edg `. + - Support for the `/TC`, `/TP`, and `-x` flags has been removed. Please ensure + all C, respectively C++, source files have a `.c`, respectively `.cpp`, + extension. + - The `--build_error_dir`, `-db`, `--edg_base_dir`, `--error_limit`, + `--src_archive`, `--trapfolder`, and `--variadic_macros` flags are now ignored. + + The above changes do not affect the creation of databases through the CodeQL CLI, + or when calling the C/C++ extractor directly with the `--mimic` or `--linker` flags. + Similar functionality continues to be supported in those scenarios, except for + CodeWarrior and the `--edg_base_dir`, `--include_path_environment`, `/Tc`, and `/Tp` + flags, which were never supported. + +### Improvements + +- `codeql generate log-summary` now reports completed pipeline runs that + are part of an incomplete recursive predicate. + +### Miscellaneous + +- The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal + documentation generation commands has been updated to version + [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). + ## Release 2.17.4 (2024-06-03) - There are no user-facing changes in this release. From ea99a263394282a49dc3a4ecacf55e738857df88 Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Tue, 18 Jun 2024 13:06:02 +0100 Subject: [PATCH 133/196] Quick fix to the 2.17.4 changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6657c0d..b44b006 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,6 +77,8 @@ ## Release 2.17.4 (2024-06-03) - There are no user-facing changes in this release. +- CodeQL package management is now generally available, and all GitHub-produced + CodeQL packages have had their version numbers increased to 1.0.0. ## Release 2.17.3 (2024-05-17) From d00e3aa63972f359ff26ced7f2cb6717d46e7a89 Mon Sep 17 00:00:00 2001 From: Florin Coada Date: Tue, 18 Jun 2024 13:29:18 +0100 Subject: [PATCH 134/196] Update CHANGELOG.md --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b44b006..dc07eb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,7 +76,8 @@ ## Release 2.17.4 (2024-06-03) -- There are no user-facing changes in this release. +### New features + - CodeQL package management is now generally available, and all GitHub-produced CodeQL packages have had their version numbers increased to 1.0.0. From ae80a68683e54c5005e12666fab8cf3a7f5e2e91 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 27 Jun 2024 19:26:34 +0200 Subject: [PATCH 135/196] Update CHANGELOG.md --- CHANGELOG.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dc07eb3..8bfe960 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,17 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.17.6 (2024-06-27) + +### New features + +- Beta support is now available for analyzing C# codebases without needing a working build. To use + this, pass the `--build-mode none` option to `codeql database create`. + +### Improvements + +- The `--model-packs` option is now publicly available. This option allows commands like `codeql database analyze` + to accept a list of model packs that are used to augment the analysis of all queries involved in the analysis. ## Release 2.17.5 (2024-06-12) @@ -335,7 +346,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 95cac33e9326a179b5d421872eac329b0b05eddb Mon Sep 17 00:00:00 2001 From: Angela P Wen Date: Thu, 11 Jul 2024 11:22:04 +0200 Subject: [PATCH 136/196] Add entry for CLI v2.18.0 --- CHANGELOG.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bfe960..0d02c47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,72 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.18.0 (2024-07-11) + +### Breaking changes + +- A number of breaking changes have been made to the C and C++ CodeQL + environment: + - The environment no longer defines any GNU-specific builtin macros. + If these macros are still needed, please define them via + `semmle-extractor-options`. + - The `--force-recompute` option is no longer directly supported by + `semmle-extractor-options`. Instead, `--edg --force-recompute` + should be specified. + - The `--gnu_version` and `--microsoft_version` options that can be + specified via `semmle-extractor-options` are now synonyms, and only + one should be specified as part of `semmle-extractor-options`. + Furthermore, is also no longer possible to specify these options + via the following syntax. + + - `--edg --gnu_version --edg `, and + - `--edg --microsoft_version --edg ` + + The shorter `--gnu_version ` and + `--microsoft_version ` should be used. +- The `--build_error_dir` and `--predefined_macros` command line options + have been removed from the C/C++ extractor. It has never been possible + to pass these options through the CLI, but some customers with advanced + setups may have been passing them through internal undocumented interfaces. + Passing the option `--build_error_dir` did not have any effect, and it + is safe to remove the option. The `--predefined_macros` option should + have been unnecessary, as long as the extractor was invoked with the + `--mimic` option. + +### Regressions + +- Compilation of QL queries is about 30% slower than in previous releases. This only affects users who write custom queries, and only at compilation time, not at run time. This regression will be fixed in the upcoming 2.18.1 release. + +### Improvements + +- Introduced the `--include-logs` option to the `codeql database bundle` + command. This new feature allows users to include logs in the generated + database bundle, allowing for a more complete treatment of the bundle, and + bringing the tool capabilities up-to-speed with the documentation. +- `codeql database init` and `codeql database create` now support the + `--force-overwrite` option. When this option is specified, the command will + delete the specified database directory even if it does not look like a + database directory. This option is only recommended for automation. For + directcommand line commands, it is recommended to use the `--overwrite` + option, which includes extra protection and will refuse to delete a + directory that does not look like a database directory. +- Extract `.xsaccess`, `*.xsjs` and `*.xsjslib` files for SAP HANA XS as + Javascript. +- We have updated many compiler error messages and warnings to improve their + readability and standardize their grammar. + Where necessary, please use the `--learn` option for the `codeql test run` + command. + +### Bugs fixed + +- Where a MacOS unsigned binary cannot be signed, CodeQL will now continue + trying to trace compiler invocations created by that process and its + children. In particular this means that Bazel builds on MacOS are now + traceable. +- Fixed a bug where test discovery would fail if there is a syntax error in a + qlpack file. Now, a warning message will be printed and discovery will + continue. + ## Release 2.17.6 (2024-06-27) ### New features From 1ff7b456c6f3386336f4b8da93caa2192d5d7ff1 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 25 Jul 2024 13:55:08 -0700 Subject: [PATCH 137/196] Update CHANGELOG.md for release 2.18.1 --- CHANGELOG.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d02c47..9644937 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,57 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.18.1 (2024-07-25) + +# Security Updates + +- Fixes CVE-2024-41807, an arbitrary file overwrite that can be triggered when + using untrusted third-party queries from a git repository. See the + [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-x4gx-f2xv-6wj9) + for more information. +- The following dependencies have been updated. These updates include + security fixes in the respective libraries that prevent + out-of-bounds accesses or denial-of-service in scenarios where + untrusted files are processed. These scenarios are not likely to be + encountered in most uses of CodeQL and code scanning, and only + apply to advanced use cases where precompiled query packs, + database ZIP files, or database TRAP files are obtained from + untrusted sources and then processed on a trusted machine. + - airlift/aircompressor is updated to version 0.27. + - Apache Ant is updated to version 1.10.11. + - Apache Commons Compress is updated to version 1.26.0. + - Apache Commons IO is updated to version 2.15.1. + - Apache Commons Lang3 is updated to version 3.14.0. + - jsoup is updated to version 1.15.3. + - Logback is updated to version 1.2.13. + - Snappy is updated to version 0.5. + +### New features + +- The *experimental* type `QlBuiltins::BigInt` of arbitrary-precision integers + has been introduced. To opt in to this API, compile your queries with + `--allow-experimental=bigint`. Big integers can be constructed using the + `.toBigInt()` methods of `int` and `string`. The built-in operations are: + - comparisons: `=`, `!=`, `<`, `<=`, `>`, `>=`, + - conversions: `.toString()`, `.toInt()`, + - arithmetic: binary `+`, `-`, `*`, `/`, `%`, unary `-`, + - bitwise operations: `.bitAnd(BigInt)`, `.bitOr(BigInt)`, + `.bitXor(BigInt)`, `.bitShiftLeft(int)`, `.bitShiftRightSigned(int)`, + `.bitNot()`, + - aggregates: `min`, `max`, (`strict`)`sum`, (`strict`)`count`, `avg`, + `rank`, `unique`, `any`. + - other: `.pow(int)`, `.abs()`, `.gcd(BigInt)`, `.minimum(BigInt)`, + `.maximum(BigInt)`. +- `codeql test run` now supports postprocessing of test results. When .qlref + files specify a path to a `postprocess` query, then this is evaluated after + the test query to transform the test outputs prior to concatenating them into + the `actual` results. + +### Improvements + +- The 30% QL query compilation slowdown noted in 2.18.0 has been fixed. + ## Release 2.18.0 (2024-07-11) ### Breaking changes @@ -412,7 +463,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From d7a1c17cba45b80adb479801425fe06452b8b1a0 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Fri, 26 Jul 2024 09:02:49 -0700 Subject: [PATCH 138/196] Update file overwrite CVE reference --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9644937..e593e5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,10 +19,11 @@ ## Release 2.18.1 (2024-07-25) -# Security Updates +### Security Updates -- Fixes CVE-2024-41807, an arbitrary file overwrite that can be triggered when - using untrusted third-party queries from a git repository. See the +- Resolves CVE-2023-4759, an arbitrary file overwrite in Eclipse JGit + that can be triggered when using untrusted third-party queries from a + git repository. See the [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-x4gx-f2xv-6wj9) for more information. - The following dependencies have been updated. These updates include From 40a98af252ec97eea890070477622cfe6d115731 Mon Sep 17 00:00:00 2001 From: Jeroen Ketema Date: Fri, 26 Jul 2024 23:25:21 +0200 Subject: [PATCH 139/196] Fix change note to clarify that C/C++ breaking changes related to tests only --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e593e5d..1ca422e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -73,10 +73,10 @@ ### Breaking changes - A number of breaking changes have been made to the C and C++ CodeQL - environment: - - The environment no longer defines any GNU-specific builtin macros. - If these macros are still needed, please define them via - `semmle-extractor-options`. + test environment as used by `codeql test run`: + - The test environment no longer defines any GNU-specific builtin + macros. If these macros are still needed by a test, please define + them via `semmle-extractor-options`. - The `--force-recompute` option is no longer directly supported by `semmle-extractor-options`. Instead, `--edg --force-recompute` should be specified. From 232a266fb3efeea4fb16b84f2038474c659de258 Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 13 Aug 2024 15:00:18 +0100 Subject: [PATCH 140/196] Update CHANGELOG for 2.18.2 --- CHANGELOG.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ca422e..1ee5fbb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,16 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.18.2 (2024-08-13) + +### Deprecations + +- Swift analysis on Ubuntu is no longer supported. Please migrate to macOS if this affects you. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.3. ## Release 2.18.1 (2024-07-25) @@ -464,7 +474,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From b601ce8efdee96a5ea7be7d42bf2a198d10cff7f Mon Sep 17 00:00:00 2001 From: Alexander Eyers-Taylor Date: Tue, 13 Aug 2024 15:01:52 +0100 Subject: [PATCH 141/196] Reintroduce Non-breaking space --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ee5fbb..5e8b14e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -474,7 +474,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 0e94992f4fbc75184f10caed298c6b93e9182c43 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Wed, 28 Aug 2024 13:46:29 +0100 Subject: [PATCH 142/196] Update changelog for 2.18.3 --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e8b14e..ba87a75 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,11 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.18.3 (2024-08-28) + +- There are no user-facing changes in this release. + ## Release 2.18.2 (2024-08-13) ### Deprecations From 961a1cbafabbf89cc0f3c06924e5361d998b0dcc Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 12 Sep 2024 10:13:15 -0400 Subject: [PATCH 143/196] Update changelog for 2.18.4 --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ba87a75..fa3e9ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.18.4 (2024-09-12) + +### New Features + +- C# support for `build-mode: none` is now out of beta, and generally available. +- Go 1.23 is now supported. + ## Release 2.18.3 (2024-08-28) - There are no user-facing changes in this release. From 79b4eacbe967fd3022bfb232feb866ef18efac2a Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Wed, 18 Sep 2024 13:50:16 +0100 Subject: [PATCH 144/196] Update CHANGELOG.md for 2.19.0 release --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index fa3e9ee..b309716 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,19 @@ you know what to do). --> +## Release 2.19.0 (2024-09-18) + +### Improvements + +- `codeql database analyze` and `codeql database interpret-results` now support + the `--sarif-run-property` option. You can provide this option when using a SARIF + output format to add a key-value pair to the property bag of the run object. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.4. + ## Release 2.18.4 (2024-09-12) ### New Features From 53387737ddf48c8d0f52d3549afe6db37f819cec Mon Sep 17 00:00:00 2001 From: Angela P Wen Date: Fri, 4 Oct 2024 13:49:36 -0700 Subject: [PATCH 145/196] Add CLI v2.19.2 changelog entry --- CHANGELOG.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b309716..a120dd2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,22 @@ you know what to do). --> +## Release 2.19.1 (2024-10-04) + +### New Features + +- The command `codeql generate query-help` now supports Markdown help files. + The Markdown help format is commonly used in custom CodeQL query packs. This new + feature allows us to generate SARIF reporting descriptors for CodeQL queries that + include Markdown help directly from a query Markdown help file. + +- Added a new command, `codeql resolve packs`. This command shows each step in the + pack search process, including what packs were found in each step. With the + `--show-hidden-packs` option, it can also show details on which packs were hidden + by packs found earlier in the search sequence. `codeql resolve packs` is intended + as a replacement for most uses of `codeql resolve qlpacks`, whose output is both + less detailed and less accurate. + ## Release 2.19.0 (2024-09-18) ### Improvements @@ -236,7 +252,7 @@ - The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal documentation generation commands has been updated to version - [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). + [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). ## Release 2.17.4 (2024-06-03) @@ -400,7 +416,7 @@ `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for - example: `codeql database create`. + example: `codeql database create`. On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. If the Python executable has a different name, you From 2e5c33400002650f7159508ab7b75eeb2b582383 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Mon, 21 Oct 2024 19:16:43 +0200 Subject: [PATCH 146/196] Update CHANGELOG.md --- CHANGELOG.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a120dd2..84e3a71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ you know what to do). --> +## Release 2.19.2 (2024-10-21) + +### Potentially breaking changes + +- The Python extractor will no longer extract the standard library by default, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. It will for a while be possible to force extraction of the standard library by setting the environment variable `CODEQL_EXTRACTOR_PYTHON_EXTRACT_STDLIB` to `1`. + +### Miscellaneous + +- The database relation `sourceLocationPrefix` is changed for databases created with + `codeql test run`. Instead of containing the path of the enclosing qlpack, it now + contains the actual path of the test, similar to if one had run `codeql database create` + on the test folder. For example, for a test such as + `/cpp/ql/test/query-tests/Security/CWE/CWE-611/XXE.qlref` we now populate + `sourceLocationPrefix` with `/cpp/ql/test/query-tests/Security/CWE/CWE-611/` + instead of `/cpp/ql/test/`. This change typically impacts calls to + `File.getRelativePath()`, and may as a result change the expected test output. + ## Release 2.19.1 (2024-10-04) ### New Features @@ -515,7 +532,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From feb1da6eeee2bd20c5e843483cdeb90366a2e46f Mon Sep 17 00:00:00 2001 From: Jonas Jensen Date: Tue, 22 Oct 2024 14:55:07 +0200 Subject: [PATCH 147/196] 2.19.2 changelog entry for evaluator bugfix --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 84e3a71..28360e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,10 @@ - The Python extractor will no longer extract the standard library by default, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. It will for a while be possible to force extraction of the standard library by setting the environment variable `CODEQL_EXTRACTOR_PYTHON_EXTRACT_STDLIB` to `1`. +### Bugs fixed + +- The 2.19.1 release contained a bug in the query evaluator that under rare conditions could lead to wrong alerts or resource exhaustion. Although we have never seen the problem outside of internal testing, we encourage users on 2.19.1 to upgrade to 2.19.2. + ### Miscellaneous - The database relation `sourceLocationPrefix` is changed for databases created with From fd9e254cb7a7240e4539cb30570e2981eb62b95f Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 7 Nov 2024 09:10:41 -0800 Subject: [PATCH 148/196] Update CHANGELOG.md for 2.19.3 release --- CHANGELOG.md | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 28360e9..d2e161f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,48 @@ you know what to do). --> +## Release 2.19.3 (2024-11-07) + +### Bugs fixed + +- Fixed a bug where using `codeql database import` to combine multiple non-empty + databases may produce a corrupted database. (The bug does not affect using + `codeql database finalize --additional-dbs` to combine multiple databases.) + +- Fixed a bug where uses of a `QlBuiltins::ExtensionId` variable that was not + bound to a value could be incorrectly accepted in some cases. In many cases, + this would result in a crash. + +- CodeQL would sometimes refuse to run with more than around 1,500 GB of RAM + available, complaining that having so much memory was "unrealistic". The + amount of memory CodeQL is able to make any meaningful use of still tops out + at about that value, but it will now gracefully accept that so large + computers do in fact exist. + +- Fixed a bug in command-line parsing where a misspelled option could sometimes + be misinterpreted as, e.g., the name of a query to run. Now every command-line + argument that begins with a dash is assumed to be intended as an option + (unless it comes after the `--` separator), and an appropriate error is + emitted if that is not a recognized one. + + The build command in `codeql database trace-command` is exempted from this for + historical reasons, but we strongly recommend putting a `--` before the entire + build command there, in case a future `codeql` version starts recognizing + options that you intended to be part of the build command. + +### Miscellaneous + +- The CodeQL Bundle is now available as an artifact that is compressed using + [Zstandard](https://en.wikipedia.org/wiki/Zstd). This artifact is + smaller and faster to decompress than the original, gzip-compressed bundle. The CodeQL bundle + is a tar archive containing tools, scripts, and various CodeQL-specific files. + + If you are currently using the CodeQL Bundle, you may want to consider switching to the + Zstandard variant of the bundle. You can download the new form of the CodeQL Bundle from the + [codeql-action releases page](https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.19.3) + by selecting the appropriate bundle with the `.zst` extension. The gzip-compressed bundles will + continue to be available for backwards compatibility. + ## Release 2.19.2 (2024-10-21) ### Potentially breaking changes @@ -536,7 +578,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 5205c9a4af4524e937df3c8a6282afcf16662359 Mon Sep 17 00:00:00 2001 From: Alex Eyers-Taylor Date: Mon, 2 Dec 2024 13:28:12 +0000 Subject: [PATCH 149/196] Add changelof for 2.19.4 --- CHANGELOG.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d2e161f..f80170c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,26 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.19.4 (2024-12-02) + +### Improvements + +- CodeQL now supports passing values containing the equals character (`=`) to extractor options via + the `--extractor-option` flag. This allows cases like `--extractor-option opt=key=value`, which + sets the extractor option `opt` to hold the value `key=value`, whereas previously that would have + been rejected with an error. +- The `codeql pack bundle` command now sets the numeric user and group IDs of entries in the generated + `tar` archive to `0`. This avoids failures like `IllegalArgumentException: user id '7111111' is too big ( > 2097151 )` + when the numeric user ID is too large. + +### Bugs fixed + +- On MacOS, `arch -arm64` commands no longer fail when they are executed via `codeql database create --command`, + via `codeql database trace-command`, or are run after `codeql database init --begin-tracing`. Note + that build commands invoked this way still will not normally be traced, so this is useful only for + running ancillary commands which are incidental to building your code. +- Fixed a bug where `codeql test run` would not preserve test + databases on disk after a test failed. ## Release 2.19.3 (2024-11-07) From 53c6f9592963d1a73bb2ea188d5687b61b6098a6 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Mon, 9 Dec 2024 17:15:25 +0000 Subject: [PATCH 150/196] Update changelog for 2.20.0 --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f80170c..58b1ca9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,20 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.0 (2024-12-09) + +### New features + +- The [`QlBuiltins::BigInt` type](https://codeql.github.com/docs/ql-language-reference/modules/#bigint) of + arbitrary precision integers is generally available and no longer hidden behind the + `--allow-experimental=bigint` CLI feature flag. + +### Miscellaneous + +- Backslashes are now escaped when writing output in the Graphviz DOT format (`--format=dot`). +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version + 21.0.5. + ## Release 2.19.4 (2024-12-02) ### Improvements From d83aedf3de84418107b73c51215d699906c605aa Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Tue, 10 Dec 2024 14:20:15 +0000 Subject: [PATCH 151/196] Add known issue for Windows exe properties version number --- CHANGELOG.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58b1ca9..f64a0b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,12 @@ --> ## Release 2.20.0 (2024-12-09) +### Known issues + +- The Windows executable for this release is labeled with an incorrect version number + within its properties: the version number should be 2.20.0 rather than 2.19.4. + `codeql version` reports the correct version number. + ### New features - The [`QlBuiltins::BigInt` type](https://codeql.github.com/docs/ql-language-reference/modules/#bigint) of @@ -612,7 +618,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From bc44bf68debfbbed6ace465d6636291545cf6b29 Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Thu, 9 Jan 2025 16:50:38 -0500 Subject: [PATCH 152/196] Update changelog for 2.20.1 --- CHANGELOG.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f64a0b8..2a1dc7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,19 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.1 (2025-01-09) + +### Improvements + +- Automatic installation of dependencies for C++ autobuild is now supported on Ubuntu 24.04. + +- The CLI will now warn if it detects that it is installed in a + location where it is likely to cause performance issues. This + includes: user home, desktop, downloads, or the file system root. + + You can avoid this warning by setting the `CODEQL_ALLOW_INSTALLATION_ANYWHERE` + environment variable to `true`. + ## Release 2.20.0 (2024-12-09) ### Known issues @@ -618,7 +631,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 1fa874136402ce0bd5ba66d4c0463ef78bd92d4b Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Wed, 22 Jan 2025 14:44:29 +0000 Subject: [PATCH 153/196] Update changelog for 2.20.2 --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a1dc7b..af98a2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,23 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.2 (2025-01-22) + +### Improvements + +- `codeql database create` and `codeql database finalize` now write relations to disk in a new, compressed format. As a result, databases will generally take up less space on disk, whether zipped or unzipped. Note that databases in this format can only be read and analyzed using CodeQL version 2.20.1 onwards. Attempting to analyze such a database with CodeQL version 2.20.0 or older will fail, with an error message like the following: + ``` + UnsortedExtensionalError: Tuples that were assumed to be in order are not: [123456777, 777654321, 123456777]<[777654321, 123456777, 777654321] + ``` + +### Enhancements + +- Added the `.bitLength()` method to `QlBuiltins::BigInt`. + +### Bugs Fixed + +- Fixed a bug where CodeQL would crash on rare occasions while merging SARIF files before uploading results. + ## Release 2.20.1 (2025-01-09) ### Improvements From 39038de5c1fe27dd47c17f81bcfc5b74eb1f8a6d Mon Sep 17 00:00:00 2001 From: Dave Bartolomeo Date: Fri, 24 Jan 2025 10:48:58 -0500 Subject: [PATCH 154/196] Changelog for 2.20.3 --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index af98a2b..e9edcbd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,19 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.20.3 (2025-01-24) + +### Security Updates + +- Resolves a security vulnerability where CodeQL databases or logs produced by the CodeQL CLI may contain the environment variables from the time of + database creation. This includes any secrets stored in an environment variables. For more information, see the + [CodeQL CLI security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gqh3-9prg-j95m). + + All users of CodeQL should follow the advice in the CodeQL advisory mentioned above or upgrade to this version or a later version of CodeQL. + + If you are using the CodeQL Action, also see the related [CodeQL Action security advisory](https://github.com/github/codeql-action/security/advisories/GHSA-vqf5-2xx6-9wfm). + ## Release 2.20.2 (2025-01-22) ### Improvements From 9a8c3477ba83a7925ceda48739e6bd3ee3063ce2 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 6 Feb 2025 12:06:35 +0100 Subject: [PATCH 155/196] Update CHANGELOG.md for 2.20.4 --- CHANGELOG.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9edcbd..5e29033 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,21 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.4 (2025-02-06) + +### New features + +- Using the `actions` language (for analysis of GitHub Actions workflows) no longer requires + the `CODEQL_ENABLE_EXPERIMENTAL_FEATURES` environment variable to be set. Support for analysis + of GitHub Actions workflows remains in public preview. + +### Bugs fixed + +- Fixed a bug where CodeQL for Java would fail with an SSL exception while trying to download `maven`. + +### Miscellaneous + +- The build of the [logback-core](https://logback.qos.ch/) library that is used for logging in the CodeQL CLI has been updated to version 1.3.15. ## Release 2.20.3 (2025-01-24) @@ -661,7 +676,7 @@ - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. - This affected the human-readable summary produced by `codeql database analyze` + This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the From 90a3e2a1b82757157110c9657fedee362c5d5575 Mon Sep 17 00:00:00 2001 From: Ian Lynagh Date: Thu, 20 Feb 2025 14:30:37 +0000 Subject: [PATCH 156/196] Add 2.20.5 to changelog --- CHANGELOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e29033..6192d4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,21 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.20.5 (2025-02-20) + +### Breaking changes + +- Removed support for `QlBuiltins::BigInt`s in the `avg()` aggregate. + +- A number of breaking changes have been made to the C and C++ CodeQL test environment as used by `codeql test run`: + - The `-Xclang-only=` option is no longer supported by `semmle-extractor-options`. Instead, when either `--clang` or `--clang_version` is specified the option should be replaced by `` only, otherwise the option should be omitted. + - The `--sys_include ` and `--preinclude ` options are no longer supported by `semmle-extractor-options`. Instead, `--edg --edg ` should be specified. + - The `-idirafter ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --sys_include --edg ` should be specified. + - The `-imacros ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --preinclude_macros --edg ` should be specified. + - The `/FI ` option is no longer supported by `semmle-extractor-options`. Instead, `--edg --preinclude --edg ` should be specified. + - The `-Wreserved-user-defined-literal`, `-Wno-reserved-user-defined-literal`, `-fwritable-strings`, `/Zc:rvalueCast`, `/Zc:rvalueCast-`, and `/Zc:wchar_t-` options are no longer supported by `semmle-extractor-options`. Instead, `--edg --reserved_user_defined_literal`, `--edg --no-reserved_user_defined_literal`, `--edg --no_const_string_literals`, `--edg --no_preserve_lvalues_with_same_type_casts`, `--edg --preserve_lvalues_with_same_type_casts`, and `--edg --no_wchar_t_keyword` should be specified, respectively. + - The `/Fo ` option is no longer supported by `semmle-extractor-options`. The option should be omitted. + ## Release 2.20.4 (2025-02-06) ### New features From a30bd7afd1834ee18b5b09a9f406cd540c10e3da Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Thu, 6 Mar 2025 07:08:35 -0800 Subject: [PATCH 157/196] Update CHANGELOG.md for 2.20.6 --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6192d4e..45b7404 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,17 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.20.6 (2025-03-06) + +### Miscellaneous + +- The CodeQL XML extractor is now able to parse documents in a wider array of + character sets. + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.6. + ## Release 2.20.5 (2025-02-20) ### Breaking changes From 87f935dfb392d8fde2421d837f2485fdcfb553ed Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Tue, 18 Mar 2025 12:37:27 +0000 Subject: [PATCH 158/196] Update CHANGELOG.md --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45b7404..37710f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.20.7 (2025-03-18) + +- There are no user-facing changes in this release. + ## Release 2.20.6 (2025-03-06) ### Miscellaneous From c22b8bd41dd90bce1a3c38253da287dfb814c59f Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 3 Apr 2025 15:43:41 +0200 Subject: [PATCH 159/196] Update CHANGELOG.md --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 37710f2..305cd8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,14 @@ you know what to do). --> +## Release 2.21.0 (2025-04-03) + +### Miscellaneous + +- On macOS the `CODEQL_TRACER_RELOCATION_EXCLUDE` environment variable can now be used to exclude certain paths from the + tracer relocation and tracing process. This environment variable accepts newline-separated regex patterns of binaries + to be excluded. + ## Release 2.20.7 (2025-03-18) - There are no user-facing changes in this release. From 518072067ef64dca25185591bf3cddd25acb3aa0 Mon Sep 17 00:00:00 2001 From: Ian Lynagh Date: Tue, 22 Apr 2025 12:33:47 +0100 Subject: [PATCH 160/196] Changelog: Update release notes for version 2.21.1 Add details about bug fixes in CodeQL analysis for GitHub Actions. Clarify behavior with `paths-ignore` and `paths` configurations to improve performance on large codebases. --- CHANGELOG.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 305cd8a..c5e2252 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,26 @@ you know what to do). --> +## Release 2.21.1 (2025-04-22) + +### Bugs fixed + +- Fixed a bug in CodeQL analysis for GitHub Actions in the presence + of a code scanning configuration file containing `paths-ignore` + exclusion patterns but not `paths` inclusion patterns. + Previously, such a configuration incorrectly led to all YAML, HTML, + JSON, and JS source files being extracted, + except for those filtered by `paths-ignore`. + This in turn led to performance issues on large codebases. + Now, only workflow and Action metadata YAML files relevant to the + GitHub Actions analysis will be extracted, + except for those filtered by `paths-ignore`. + This matches the default behavior when no configuration file + is provided. + The handling of `paths` inclusion patterns is unchanged: + if provided, only those paths will be considered, + except for those filtered by `paths-ignore`. + ## Release 2.21.0 (2025-04-03) ### Miscellaneous From d1e534e60eafaca3770361ddcfc6b53a06068d91 Mon Sep 17 00:00:00 2001 From: Nick Rolfe Date: Thu, 1 May 2025 13:34:25 +0100 Subject: [PATCH 161/196] Update changelog for 2.21.2 --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c5e2252..2a9962a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.21.2 (2025-05-01) + +### Bugs fixed + +- `codeql generate log-summary` now correctly includes `dependencies` + maps in predicate events for `COMPUTED_EXTENSIONAL` predicates. + ## Release 2.21.1 (2025-04-22) ### Bugs fixed From 9a367b9f8833da440fdde706789c18cb11f21624 Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 15 May 2025 12:41:58 +0100 Subject: [PATCH 162/196] Update changelog for 2.21.3 --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a9962a..9f8780a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.21.3 (2025-05-15) + +### Miscellaneous + +- Windows binaries for the CodeQL CLI are now built with `/guard:cf`, enabling [Control Flow Guard](https://learn.microsoft.com/en-us/windows/win32/secbp/control-flow-guard). + ## Release 2.21.2 (2025-05-01) ### Bugs fixed From f1c6d0410fc75afe3b3c146bee902cbd83e19fb8 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Mon, 2 Jun 2025 13:43:16 +0200 Subject: [PATCH 163/196] Update CHANGELOG.md --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f8780a..df6571f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ you know what to do). --> +## Release 2.21.4 (2025-06-02) + +### Deprecations + +- The `clang_vector_types`, `clang_attributes`, and `flax-vector-conversions` command + line options have been removed from the C/C++ extractor. These options were introduced + as workarounds to frontend limitations in earlier versions of the extractor and are + no longer needed when calling the extractor directly. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.7. + ## Release 2.21.3 (2025-05-15) ### Miscellaneous From 81e6755f40ec81a3583d4e2ba929819b0b0fc375 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Wed, 11 Jun 2025 08:40:32 -0700 Subject: [PATCH 164/196] Update CHANGELOG.md for 2.22.0 --- CHANGELOG.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index df6571f..48b2c6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,28 @@ you know what to do). --> +## Release 2.22.0 (2025-06-11) + +### Breaking changes + +- A number of breaking changes have been made to the C and C++ CodeQL test + environment as used by `codeql test run`: + - Options starting with a `/` are no longer supported by + `semmle-extractor-options`. Any option starting with a `/` should be + replaced by the equivalent option starting with a `-`, e.g., `/D` should be + replaced by `-D`. + - Preprocessor command line options of the form `-D#` are no + longer supported by `semmle-extractor-options`. `-D=` should be + used instead. + - The `/Fp` and `-o` options are no longer supported by + `semmle-extractor-options`. The options should be omitted. + - The `-emit-pch`, `-include-pch`, `/Yc`, and `/Yu` options, and the + `--preinclude` option taking a pre-compiled header as its argument, are no + longer supported by `semmle-extractor-options`. Any test that makes use of + this should be replaced by a test that invokes the CodeQL CLI with the + `create database` option and that runs the relevant queries on the created + database. + ## Release 2.21.4 (2025-06-02) ### Deprecations From 37a1db679b0bbb4e51158553a88d9bfa000efdf2 Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 26 Jun 2025 13:08:22 +0200 Subject: [PATCH 165/196] Update CHANGELOG.md for 2.22.1 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 48b2c6e..925d692 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.22.1 (2025-06-26) + +### New features + +- Rust language support is now in public preview. + +### Miscellaneous + +- The version of `jgit` used by the CodeQL CLI has been updated to `6.10.1.202505221210-r`. + ## Release 2.22.0 (2025-06-11) ### Breaking changes From a744fb1decf08c0ba3b90f4944b8019c94bee4f5 Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Tue, 29 Jul 2025 10:00:27 -0700 Subject: [PATCH 166/196] Update CHANGELOG.md for 2.22.2 --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 925d692..013ee91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,17 @@ you know what to do). --> +## Release 2.22.2 (2025-07-29) + +### Bug fix + +- Fixes a bug in query suites where the `version` property of an `import` instruction was ignored. Previously, the following query suite would _not_ resolve to `v1.0.19` of `codeql/csharp-queries`. Instead it would resolve to the latest version. This is now fixed and the resolve pack version would be `v1.0.19`. + ``` + - from: codeql/csharp-queries + import: codeql-suites/csharp-security-and-quality.qls + version: 1.0.19 + ``` + ## Release 2.22.1 (2025-06-26) ### New features From d2abcd0678d06f9bb0375bc74e0bb4bda392014b Mon Sep 17 00:00:00 2001 From: Chuan-kai Lin Date: Wed, 6 Aug 2025 13:28:32 -0700 Subject: [PATCH 167/196] Update CHANGELOG.md for 2.22.3 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 013ee91..6ee09a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,14 @@ you know what to do). --> +## Release 2.22.3 (2025-08-06) + +### New features + +- The `codeql database cleanup` command now takes the `--cache-cleanup=overlay` + option, which trims the cache to just the data that will be useful when + evaluating against an overlay. + ## Release 2.22.2 (2025-07-29) ### Bug fix From 99380d934e6d13b263ce8d4bc8b62f51369c158b Mon Sep 17 00:00:00 2001 From: Chris Smowton Date: Thu, 21 Aug 2025 11:48:12 +0100 Subject: [PATCH 168/196] Update changelog for 2.22.4 release --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ee09a5..bc43e08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ you know what to do). --> +## Release 2.22.4 (2025-08-21) + +- There are no user-facing changes in this release. + ## Release 2.22.3 (2025-08-06) ### New features From 108cd2005eae9cc5899871aa00aac279b3a8d3be Mon Sep 17 00:00:00 2001 From: Arthur Baars Date: Thu, 4 Sep 2025 17:01:49 +0200 Subject: [PATCH 169/196] Update CHANGELOG.md --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc43e08..441b99c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.23.0 (2025-09-04) + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.8. + ## Release 2.22.4 (2025-08-21) - There are no user-facing changes in this release. From 56734edf29be6e0363e1a583ecb622c0b0b2476e Mon Sep 17 00:00:00 2001 From: Ian Lynagh Date: Tue, 23 Sep 2025 14:37:48 +0100 Subject: [PATCH 170/196] Update CHANGELOG.md for 2.23.1 --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 441b99c..f559444 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ you know what to do). --> +## Release 2.23.1 (2025-09-23) + +### New features + +- CodeQL now adds the sources and sinks of path alerts to the `relatedLocations` + property of SARIF results if they are not included as the primary location or + within the alert message. This means that path alerts will show on PRs if a + source or sink is added or modified, even for queries that don't follow the + common convention of selecting the sink as the primary location and mentioning + the source in the alert message. + +- CodeQL now populates file coverage information for GitHub Actions on + [the tool status page for code scanning](https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/about-the-tool-status-page#viewing-the-tool-status-page-for-a-repository). + ## Release 2.23.0 (2025-09-04) ### Miscellaneous From 194e5fbf3a17ee992ee5aa145e5ef43c719caa99 Mon Sep 17 00:00:00 2001 From: Nick Rolfe Date: Thu, 2 Oct 2025 11:27:58 +0100 Subject: [PATCH 171/196] Changelog for 2.32.2 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f559444..ed6b7d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.23.2 (2025-10-02) + +### New features + +- CodeQL Go analysis now supports the "Git Source" type for [private package registries](https://docs.github.com/en/code-security/securing-your-organization/enabling-security-features-in-your-organization/giving-org-access-private-registries). This is in addition to the existing support for the "GOPROXY server" type. + +### Fixes + +- The `codeql generate query-help` command now prepends the query's name (taken from the `.ql` file) as a level-one heading when processing markdown query help, for consistency with help generated from a `.qhelp` file. + ## Release 2.23.1 (2025-09-23) ### New features From 05e07ac5e52e3f77e1f77105f02437f982ada263 Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Fri, 17 Oct 2025 14:20:03 +0100 Subject: [PATCH 172/196] Update CHANGELOG.md for 2.23.3 --- CHANGELOG.md | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed6b7d4..704643a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,13 +17,40 @@ you know what to do). --> +## Release 2.23.3 (2025-10-17) + +### Breaking changes + +- The `--permissive` command line option has been removed from the C/C++ extractor, + and passing the option will make the extractor fail. The option was introduced to + make the extractor accept the following invalid code, which is accepted by gcc with + the `-fpermissive` flag: + + ```cpp + void f(char*); + void g() { + const char* str = "string"; + f(str); + } + ``` + + The `--permissive` option was removed, as under some circumstances it would break the extractor's ability to parse valid C++ code. When calling the extractor directly, + `--permissive` should no longer be passed. The above code will fail to parse, and we + recommend the code being made `const`-correct. + +### Bugs fixed + +- Fixed a bug that made many `codeql` subcommands fail with the + message `not in while, until, select, or repeat loop` on Linux or + macOS systems where `/bin/sh` is `zsh`. + ## Release 2.23.2 (2025-10-02) ### New features - CodeQL Go analysis now supports the "Git Source" type for [private package registries](https://docs.github.com/en/code-security/securing-your-organization/enabling-security-features-in-your-organization/giving-org-access-private-registries). This is in addition to the existing support for the "GOPROXY server" type. -### Fixes +### Bugs Fixed - The `codeql generate query-help` command now prepends the query's name (taken from the `.ql` file) as a level-one heading when processing markdown query help, for consistency with help generated from a `.qhelp` file. From bbcf258d7eb900825504fb234aa27111119a895a Mon Sep 17 00:00:00 2001 From: "Michael B. Gale" Date: Thu, 6 Nov 2025 17:02:41 +0000 Subject: [PATCH 173/196] Add permissions to workflow Also update workflow a bit --- .github/workflows/label-issue.yml | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/.github/workflows/label-issue.yml b/.github/workflows/label-issue.yml index 9c2567c..7494234 100644 --- a/.github/workflows/label-issue.yml +++ b/.github/workflows/label-issue.yml @@ -6,11 +6,14 @@ on: jobs: label: name: Label issue - runs-on: ubuntu-latest - if: github.event.action == 'opened' + runs-on: ubuntu-slim + permissions: + issues: write steps: - name: Label issue + run: gh issue edit "$NUMBER" --add-label "$LABELS" env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - echo '{"labels": ["CLI"]}' | gh api repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/labels --input - + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GH_REPO: ${{ github.repository }} + NUMBER: ${{ github.event.issue.number }} + LABELS: CLI From f66af530a4da7876739cd1d7d18f8ad989ea8757 Mon Sep 17 00:00:00 2001 From: "Michael B. Gale" Date: Thu, 13 Nov 2025 20:55:46 +0000 Subject: [PATCH 174/196] Update CHANGELOG.md for 2.23.5 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 704643a..69b2c36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.23.5 (2025-11-13) + +### Breaking changes + +- In order to make a `@kind path-problem` query diff-informed, the `getASelectedSourceLocation` and `getASelectedSinkLocation` predicates in the dataflow configuration now need to be overridden to always return the location of the source/sink _in addition to_ any other locations that are selected by the query. See the [QLdoc](https://github.com/github/codeql/blob/d122534398c5eb9182a23a9ad65caa5937d627b5/shared/dataflow/codeql/dataflow/DataFlow.qll#L474) for more details. + +## Release 2.23.4 + +This release was skipped. + ## Release 2.23.3 (2025-10-17) ### Breaking changes From 7fa26143c5fe317c7c0e6e9e16be8c8c14652dbd Mon Sep 17 00:00:00 2001 From: Paolo Tranquilli Date: Mon, 24 Nov 2025 09:38:34 +0100 Subject: [PATCH 175/196] Update CHANGELOG for release 2.23.6 --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 69b2c36..9fb7cb4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.23.6 (2025-11-24) + +### Breaking changes + +- The LGTM results format for uploading to LGTM has been removed. + ## Release 2.23.5 (2025-11-13) ### Breaking changes From 2dd2c45a547668c46ec8a7b7c41af88cb024974e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Fri, 5 Dec 2025 15:26:00 +0100 Subject: [PATCH 176/196] Document deprecation of '--save-cache' flag Added deprecation notice for the '--save-cache' flag. --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9fb7cb4..28f1d63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.23.7 (2025-12-05) + +### Deprecations + +- The `--save-cache` flag to `codeql database run-queries` and other commands that execute queries has been deprecated. This flag previously instructed the evaluator to aggressively write intermediate results to the disk cache, but now has no effect. + ## Release 2.23.6 (2025-11-24) ### Breaking changes @@ -3995,3 +4001,4 @@ become available. ## Release 2.0.0 (2019-11-14) - First public release. + From 998e37ce3edfb2ea5ddf4c4f1b91ea5bd5733a73 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Thu, 11 Dec 2025 17:34:07 +0100 Subject: [PATCH 177/196] Document release 2.23.8 in CHANGELOG.md Added release notes for version 2.23.8. --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 28f1d63..62520e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,9 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.23.8 (2025-12-10) + +This release contains no CLI changes. ## Release 2.23.7 (2025-12-05) From fe40a78563ac3c0a2b6c621984bafdced2dba969 Mon Sep 17 00:00:00 2001 From: Ian Lynagh Date: Fri, 9 Jan 2026 17:29:47 +0000 Subject: [PATCH 178/196] Update CHANGELOG.md for 2.23.9 --- CHANGELOG.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62520e9..a6ea842 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,13 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.23.9 (2026-01-09) + +### Deprecations + +- Support for Kotlin version 1.6 and 1.7 has been deprecated and will be removed from CodeQL version 2.24.1. Starting with version 2.24.1, users will need to use Kotlin version >= 1.8 to extract Kotlin databases. + ## Release 2.23.8 (2025-12-10) This release contains no CLI changes. @@ -4004,4 +4011,3 @@ become available. ## Release 2.0.0 (2019-11-14) - First public release. - From 6866fd9f6579dd1f86af647a08790a77a2f6d1b3 Mon Sep 17 00:00:00 2001 From: Nick Rolfe Date: Mon, 26 Jan 2026 12:42:45 +0000 Subject: [PATCH 179/196] Add changelog for 2.24.0 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a6ea842..dad374a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,16 @@ you know what to do). --> +## Release 2.24.0 (2026-01-26) + +### Miscellaneous + +- The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal + documentation generation commands has been updated to version + [20260102.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20260102.1). +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.9. + ## Release 2.23.9 (2026-01-09) ### Deprecations From c9e1ebc8d88ba207ca596a120fbae1cecd3c47f5 Mon Sep 17 00:00:00 2001 From: Nathan Randall Date: Mon, 2 Feb 2026 09:16:36 -0700 Subject: [PATCH 180/196] Fix typos in LICENSE.md This commit fixes the spelling of one word and corrects one subject-verb agreement mismatch in LICENSE.md file. --- LICENSE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE.md b/LICENSE.md index 3ef032d..e872708 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -33,7 +33,7 @@ below: * Use the Software to demonstrate the Software. * Test CodeQL queries that are released under an OSI-approved - Licence to confirm that new versions of those queries continue to + License to confirm that new versions of those queries continue to find the right vulnerabilities. Here's what you may also do with the Software, but only with an Open @@ -169,7 +169,7 @@ provision of these Terms will not constitute a waiver of such right or provision. _Entire Agreement._ These Terms, together with any open source -software licenses referenced above, constitutes the entire agreement +software licenses referenced above, constitute the entire agreement between you and GitHub regarding your use of the Software, superseding any prior agreements between you and GitHub (including, but not limited to, any prior versions of these Terms) regarding such use. From 3f1fd5f017d87e1f956b5a8d877ffd8cb4a5176d Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Thu, 5 Feb 2026 15:56:12 +0000 Subject: [PATCH 181/196] Add changenotes for 2.24.1 --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index dad374a..a636b80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.24.1 (2026-02-05) + +### Miscellaneous + +- The vulnerable xwork-core 2.3.37 test dependency (CVE-2025-68493) has been removed. The CodeQL Java library has been updated to support both legacy Struts 2.x-6.x package names and Struts 7.x package names for analyzing user code. + ## Release 2.24.0 (2026-01-26) ### Miscellaneous From f4e1dee21aedb8d382f6ac4a3dfb514a8e4d0e3b Mon Sep 17 00:00:00 2001 From: "Michael B. Gale" Date: Fri, 20 Feb 2026 11:20:13 +0000 Subject: [PATCH 182/196] Update CHANGELOG.md for 2.24.2 --- CHANGELOG.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a636b80..916665c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.24.2 (2026-02-20) + +### Bug Fixes + +- Fixed SARIF output to generate RFC 1738 compatible file URIs. File URIs now always use the `file:///` format instead of `file:/` for better interoperability with SARIF consumers. + ## Release 2.24.1 (2026-02-05) ### Miscellaneous @@ -82,7 +88,7 @@ This release was skipped. } ``` - The `--permissive` option was removed, as under some circumstances it would break the extractor's ability to parse valid C++ code. When calling the extractor directly, + The `--permissive` option was removed, as under some circumstances it would break the extractor's ability to parse valid C++ code. When calling the extractor directly, `--permissive` should no longer be passed. The above code will fail to parse, and we recommend the code being made `const`-correct. @@ -231,7 +237,7 @@ This release was skipped. - On macOS the `CODEQL_TRACER_RELOCATION_EXCLUDE` environment variable can now be used to exclude certain paths from the tracer relocation and tracing process. This environment variable accepts newline-separated regex patterns of binaries - to be excluded. + to be excluded. ## Release 2.20.7 (2025-03-18) From 9231df8f23df70b6150bd79223c42d5e20c94934 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Thu, 5 Mar 2026 17:08:45 +0100 Subject: [PATCH 183/196] Update CHANGELOG.md for 2.24.3 From f467d25f90410a13ab54482a71c991714ac8a3a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Thu, 5 Mar 2026 17:10:54 +0100 Subject: [PATCH 184/196] Update CHANGELOG for release 2.24.3 Added details about bug fixes in release 2.24.3, including race condition fix and spurious warnings. --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 916665c..e429b8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,13 @@ you know what to do). --> +## Release 2.24.3 (2026-03-05) + +### Bug Fixes + +- Fixed a race condition that could cause flaky failures in overlay CodeQL tests. Test extraction now skips `*.testproj` directories by name, preventing interference from concurrently cleaned-up test databases. +- Fixed spurious "OOPS" warnings that could appear in help output for commands using mutually exclusive option groups, such as `codeql query run`. + ## Release 2.24.2 (2026-02-20) ### Bug Fixes From 249f3d5b0553b743286d11218ae16297c13a9cc4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Thu, 19 Mar 2026 13:15:33 +0100 Subject: [PATCH 185/196] Update CHANGELOG for release 2.25.0 Added release notes for version 2.25.0, including breaking changes and bug fixes. --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e429b8c..489bd40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,20 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.25.0 (2026-03-19) + +### Breaking Changes + +- `codeql database interpret-results` and `codeql database analyze` no longer attempt to reconstruct file baseline information from databases created with CLI versions before 2.11.2. + +### Bug Fixes + +- Upgraded Jackson library from 2.16.1 to 2.18.6 to address a high-severity denial of service vulnerability (GHSA-72hv-8253-57qq) in jackson-core's async JSON parser. +- Upgraded snakeyaml (which is a dependency of jackson-dataformat-yaml) from 2.2 to 2.3. + +## Release 2.24.4 (2026-03-16) + +This release was skipped. ## Release 2.24.3 (2026-03-05) From 1ebe9749ba4d4fc7a4d4f4cb900c8c2fd6a52893 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Fri, 27 Mar 2026 09:44:04 +0000 Subject: [PATCH 186/196] Update changelog for release 2.25.1 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 489bd40..4f60231 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,16 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.25.1 (2026-03-27) + +### Bug Fixes + +- Fixed a bug where extraction could fail on YAML files containing emoji.Collapse commentComment on lines R24 to R25henrymercer commented on Mar 26, 2026 henrymerceron Mar 26, 2026More actions + +### Miscellaneous + +- Upgraded snakeyaml (which is a dependency of jackson-dataformat-yaml) from 2.3 to 2.6. + ## Release 2.25.0 (2026-03-19) ### Breaking Changes From 0564862bef092139fbd761ee56dc3f2380555ef5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Fri, 27 Mar 2026 10:49:07 +0100 Subject: [PATCH 187/196] Fix YAML extraction bug and upgrade snakeyaml --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f60231..90fb0e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ ### Bug Fixes -- Fixed a bug where extraction could fail on YAML files containing emoji.Collapse commentComment on lines R24 to R25henrymercer commented on Mar 26, 2026 henrymerceron Mar 26, 2026More actions +- Fixed a bug where extraction could fail on YAML files containing emoji. ### Miscellaneous From 2c725ac18f6b371750d317f5f98d247c75d4c85b Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Wed, 15 Apr 2026 11:44:22 +0100 Subject: [PATCH 188/196] Add changenotes for 2.25.2 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 90fb0e7..0e93e38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,14 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.25.2 (2026-04-15) + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.10. + ## Release 2.25.1 (2026-03-27) ### Bug Fixes From 52038d2120bcc5565fa406ca116f901efaa10f00 Mon Sep 17 00:00:00 2001 From: "Michael B. Gale" Date: Fri, 1 May 2026 13:05:14 +0100 Subject: [PATCH 189/196] Add changenotes for 2.25.3 --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e93e38..0e2b723 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,17 @@ you know what to do). --> +## Release 2.25.3 (2026-05-01) + +### Improvements + +- The `codeql database finalize` command now accepts the `--working-dir` + flag. When specified, any extractor pre-finalize scripts will be run in + that directory. If the flag is not used, the scripts will run in the source + root directory (maintaining existing behavior). The flag will also be + automatically passed through when running the higher-level + `codeql database create` command. + ## Release 2.25.2 (2026-04-15) ### Miscellaneous From caa82085d08289b56d2f6fea87812e4115c459e4 Mon Sep 17 00:00:00 2001 From: Paolo Tranquilli Date: Thu, 7 May 2026 15:50:25 +0200 Subject: [PATCH 190/196] Add CodeQL 2.25.4 to CHANGELOG Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e2b723..1d9dc59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,11 @@ you know what to do). --> -## Release 2.25.3 (2026-05-01) +## Release 2.25.4 (2026-05-05) + +- There are no user-facing changes in this release. + +## Release 2.25.3 (2026-04-30) ### Improvements From 697ca25a6968ae01bab1b11ae56c3be5960f588c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Fri, 22 May 2026 12:32:52 +0200 Subject: [PATCH 191/196] Update CHANGELOG.md --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d9dc59..1eba48c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,9 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.25.5 (2026-05-21) + +- There are no user-facing changes in this release. ## Release 2.25.4 (2026-05-05) From 09b86de42c26199a1e59caa9f5ad5c81384a3c0d Mon Sep 17 00:00:00 2001 From: Henry Mercer Date: Thu, 4 Jun 2026 12:03:17 +0100 Subject: [PATCH 192/196] Update changelog for release 2.25.6 --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1eba48c..e127ee0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,19 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.25.6 (2026-06-04) + +### Improvements + +- When the `git` executable is available, CodeQL can now obtain configuration and + queries from SHA-256 Git repositories, and infer Git metadata about them. + +### Miscellaneous + +- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL + CLI has been updated to version 21.0.11. + ## Release 2.25.5 (2026-05-21) - There are no user-facing changes in this release. From 39d9d4aa59afb4ddf710a09ea6a14c9f918fa35d Mon Sep 17 00:00:00 2001 From: "Michael B. Gale" Date: Wed, 8 Jul 2026 08:44:08 +0100 Subject: [PATCH 193/196] Update changelog for 2.26.0 --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e127ee0..c34d194 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,12 @@ you know what to do). --> +## Release 2.26.0 (2026-07-08) + +### Improvements + +- Improved the performance of commands that interact with Git repositories by checking whether the `git` command-line tool is available at most once per CodeQL CLI invocation. + ## Release 2.25.6 (2026-06-04) ### Improvements From 865c3ee15ca1080c18d17b5c8df3b53c16e8aa3f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Thu, 16 Jul 2026 11:31:03 +0200 Subject: [PATCH 194/196] Update CHANGELOG.md --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c34d194..c52194b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,9 @@ you know what to do). --> +## Release 2.26.1 (2026-07-15) + +- There are no user-facing changes in this release. ## Release 2.26.0 (2026-07-08) ### Improvements From 1c54fd8be7e39ed654fdd1f5999aa3320c7edb34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=93scar=20San=20Jos=C3=A9?= Date: Wed, 29 Jul 2026 16:18:33 +0200 Subject: [PATCH 195/196] Document breaking change in release 2.26.2 Removed support for parsing [[-style links in alert messages, replacing it with $@ placeholder pairs. --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c52194b..272a8cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,11 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> +## Release 2.26.2 (2026-07-23) + +### Breaking Changes + +- Removed support for parsing `[[`-style links in alert messages. This was an undocumented legacy feature that allowed query authors to embed links inline in select clause message strings using `[["text"|"url"]]` syntax. Queries should use `$@` placeholder pairs instead. ## Release 2.26.1 (2026-07-15) From 0d65148c254764ec294892a35e644accd5677ed5 Mon Sep 17 00:00:00 2001 From: "Michael B. Gale" Date: Wed, 12 Aug 2026 16:53:31 +0100 Subject: [PATCH 196/196] Update changelog for 2.26.3 --- CHANGELOG.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 272a8cf..0934aee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,22 @@ checklist for a CLI release, you can edit here. But then you know what to do). --> + +## Release 2.26.3 (2026-08-12) + +### Improvements + +- Commands that accept a `--ram` option now report a clear error when given a value that is far too large to be a sensible amount of memory in megabytes (for example, a number of bytes passed by mistake), instead of failing with a cryptic "is not an int" message. + +### Bug Fixes + +- Fixed the pack upload format to comply with the OCI-image-manifest specification, by ensuring that all required fields are present/valid, and that no extraneous/non-compliant fields are included. +- Fixed path canonicalization on Windows so that paths located on `subst`ed drives are always resolved to their underlying target paths. Previously, `subst`ed drives were not handled consistently by the CodeQL CLI and the language-specific extractors. + +### Miscellaneous + +- Upgraded Jackson from 2.18.6 to 2.18.9. + ## Release 2.26.2 (2026-07-23) ### Breaking Changes