diff --git a/CHANGELOG.md b/CHANGELOG.md index 24659e0d240..50d888acbd3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,10 @@ # Changelog for Elixir v1.17 +This release includes type inference of patterns to provide warnings for an initial set of constructs (binaries, maps, and atoms) within the same function. It also includes a new Duration data type to interact with Calendar types, support for Erlang/OTP 27, and many other improvements. + ## Warnings from gradual set-theoretic types -This release introduces gradual set-theoretic types to infer types from patterns and guards and use them to type check programs, enabling the Elixir compiler to find faults and bugs in codebases without requiring changes to existing software. The underlying principles, theory, and roadmap of our work have been outlined in ["The Design Principles of the Elixir Type System" by Giuseppe Castagna, Guillaume Duboc, José Valim](https://arxiv.org/abs/2306.06391). +This release introduces gradual set-theoretic types to infer types from patterns and use them to type check programs, enabling the Elixir compiler to find faults and bugs in codebases without requiring changes to existing software. The underlying principles, theory, and roadmap of our work have been outlined in ["The Design Principles of the Elixir Type System" by Giuseppe Castagna, Guillaume Duboc, José Valim](https://arxiv.org/abs/2306.06391). At the moment, Elixir developers will interact with set-theoretic types only through warnings found by the type system. The current implementation models all data types in the language: @@ -10,7 +12,7 @@ At the moment, Elixir developers will interact with set-theoretic types only thr * `atom()` - it represents all atoms and it is divisible. For instance, the atom `:foo` and `:hello_world` are also valid (distinct) types. - * `map()` and structs - maps can be "closed" or "open". Closed maps only allow the specified allows keys, such as `%{key: atom(), value: integer()}`. Open maps support any other keys in addition to the ones listed and their definition starts with `...`, such as `%{..., key: atom(), value: integer()}`. Structs are closed maps with the `__struct__` key. + * `map()` and structs - maps can be "closed" or "open". Closed maps only allow the specified keys, such as `%{key: atom(), value: integer()}`. Open maps support any other keys in addition to the ones listed and their definition starts with `...`, such as `%{..., key: atom(), value: integer()}`. Structs are closed maps with the `__struct__` key. * `tuple()`, `list()`, and `function()` - currently they are modelled as indivisible types. The next Elixir versions will also introduce fine-grained support to them. @@ -20,25 +22,23 @@ We focused on atoms and maps on this initial release as they are respectively th * Accessing a key on a map or a struct that does not have the given key, such as `user.adress` - * Updating a struct or a map that does not define the given key, such as `%{user | adress: ...}` - * Invoking a function on non-modules, such as `user.address()` * Capturing a function on non-modules, such as `&user.address/0` + * Attempting to invoke to call an anonymous function without an actual function, such as `user.()` + * Performing structural comparisons with structs, such as `my_date < ~D[2010-04-17]` * Performing structural comparisons between non-overlapping types, such as `integer >= string` - * Building and pattern matching on binaries without the relevant specifiers, such as `<>` (this warns because by default it expects an integer) + * Building and pattern matching on binaries without the relevant specifiers, such as `<>` (this warns because by default it expects an integer, it should have been `<>` instead) - * Attempting to rescue an undefined exception or an exception that is not a struct + * Attempting to rescue an undefined exception or a struct that is not an exception * Accessing a field that is not defined in a rescued exception -These new warnings help Elixir developers find bugs earlier and give more confidence when refactoring code, especially around maps and structs. While some of these warnings were emitted in the past, they were discovered using syntax analysis. The new warnings are more reliable, precise, and with better error messages. Keep in mind that not all maps have statically known keys, and the Elixir typechecker only infers types from patterns at the moment. - -Future Elixir versions will infer and type check more constructs, bringing Elixir developers more warnings and quality of life improvements without changes to code. For more details, see our new [reference document on gradual set-theoretic types](https://hexdocs.pm/elixir/main/gradual-set-theoretic-types.html). +These new warnings help Elixir developers find bugs earlier and give more confidence when refactoring code, especially around maps and structs. While some of these warnings were emitted in the past, they were discovered using syntax analysis. The new warnings are more reliable, precise, and with better error messages. Keep in mind, however, that the Elixir typechecker only infers types from patterns within the same function at the moment. Analysis from guards and across function boundaries will be added in future relases. For more details, see our new [reference document on gradual set-theoretic types](https://hexdocs.pm/elixir/gradual-set-theoretic-types.html). The type system was made possible thanks to a partnership between [CNRS](https://www.cnrs.fr/) and [Remote](https://remote.com/). The development work is currently sponsored by [Fresha](https://www.fresha.com/), [Starfish*](https://starfish.team/), and [Dashbit](https://dashbit.co/). @@ -70,7 +70,62 @@ Finally, a new `Kernel.to_timeout/1` function has been added, which helps develo Process.send_after(pid, :wake_up, to_timeout(hour: 1)) ``` -## v1.17.0-dev +## v1.17.3 (2024-09-18) + +### 1. Bug fixes + +#### Elixir + + * [Duration] Fix parsing of fractional durations with non-positive seconds + * [Kernel] Do not attempt to group module warnings when they have a large context + +#### IEx + + * [IEx.Helpers] Properly reconsolidate protocols on `recompile` + +#### Mix + + * [mix compile.elixir] Do not verify modules twice + * [mix xref] Respect the `--label` option on stats and cycles + +## v1.17.2 (2024-07-06) + +### 1. Bug fixes + +#### Logger + + * [Logger.Translator] Fix logger crash when `:gen_statem`'s `format_status/2` returns non-tuple + +#### Mix + + * [mix deps.get] Fix regression when fetching a git repository with a `:ref` + * [mix release] Validate `RELEASE_MODE` and set ERRORLEVEL on `.bat` scripts + * [mix release] Fix invalid example in code comment inside the generated vm.args.eex + +## v1.17.1 (2024-06-18) + +### 1. Enhancements + +#### Mix + + * [mix compile.elixir] Do not run fixpoint computation on runtime dependencies. This should considerably improve compilation times for large projects when changing only one or two files + +### 2. Bug fixes + +#### EEx + + * [EEx] Do not warn for assignment with blocks in EEx + +#### Elixir + + * [Kernel] Fix bug when using pinned variables inside `with`'s `else` patterns + * [Kernel] Fix Dialyzer error when with else clause is calling a `no_return` function + +#### ExUnit + + * [ExUnit] Do not alternative sync/async suites on `--repeat-until-failure` + +## v1.17.0 (2024-06-12) ### 1. Enhancements @@ -88,6 +143,7 @@ Process.send_after(pid, :wake_up, to_timeout(hour: 1)) * [Kernel] Add `Kernel.to_timeout/1` * [Kernel] Emit warnings for undefined functions from modules defined within the same context as the caller code * [Kernel] Support integers in uppercase sigils + * [Keyword] Add `Keyword.intersect/2-3` to mirror the `Map` API * [Macro] Add `Macro.Env.define_alias/4`, `Macro.Env.define_import/4`, `Macro.Env.define_require/4`, `Macro.Env.expand_alias/4`, `Macro.Env.expand_import/5`, and `Macro.Env.expand_require/6` to aid the implementation of language servers and embedded languages * [NaiveDateTime] Add `NaiveDateTime.shift/2` to shift naive datetimes with duration and calendar-specific semantics * [Process] Add `Process.set_label/1` @@ -117,9 +173,11 @@ Process.send_after(pid, :wake_up, to_timeout(hour: 1)) * [mix deps] Warn if `:optional` is used in combination with `:in_umbrella` * [mix deps.get] Do not add optional dependency requirements if its parent dep was skipped * [mix deps.tree] Add `--umbrella-only` to `mix deps.tree` + * [mix profile.tprof] Add a new profiler, available on Erlang/OTP 27+, which can measure count, time, and heap usage * [mix test] Add `mix test --breakpoints` that sets up a breakpoint before each test that will run * [mix test] Add `mix test --repeat-until-failure` to rerun tests until a failure occurs * [mix test] Add `mix test --slowest-modules` to print slowest modules based on all of the tests they hold + * [mix test] Generate cover HTML files in parallel ### 2. Bug fixes @@ -132,7 +190,10 @@ Process.send_after(pid, :wake_up, to_timeout(hour: 1)) * [Kernel] Do not expand code in `quote bind_quoted: ...` twice * [Kernel] Respect `:line` property when `:file` is given as option to `quote` * [Kernel] Do not crash on `Macro.escape/2` when passing a quote triplet without valid meta + * [Kernel] Avoid double tracing events when capturing a function + * [Kernel] Fix a bug where captured arguments would conflict when a capture included a macro that also used captures * [Module] Return default value in `Module.get_attribute/3` for persisted attributes which have not yet been written to + * [String] Properly handle transpositions in `jaro_distance`. This will correct the distance result in certain cases #### IEx @@ -148,6 +209,11 @@ Process.send_after(pid, :wake_up, to_timeout(hour: 1)) * [GenServer] Deprecate `c:GenServer.format_status/2` callback to align with Erlang/OTP 25+ +#### Mix + + * [mix profile.cprof] Deprecated in favor of the new `mix profile.tprof` + * [mix profile.eprof] Deprecated in favor of the new `mix profile.tprof` + ### 4. Hard deprecations #### Elixir diff --git a/Makefile b/Makefile index da0caa1ec92..2cac709428b 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ PREFIX ?= /usr/local TEST_FILES ?= "*_test.exs" SHARE_PREFIX ?= $(PREFIX)/share MAN_PREFIX ?= $(SHARE_PREFIX)/man -CANONICAL := main/ +# CANONICAL := main/ ELIXIRC := bin/elixirc --ignore-module-conflict $(ELIXIRC_OPTS) ERLC := erlc -I lib/elixir/include ERL_MAKE := if [ -n "$(ERLC_OPTS)" ]; then ERL_COMPILER_OPTIONS=$(ERLC_OPTS) erl -make; else erl -make; fi diff --git a/RELEASE.md b/RELEASE.md index 42ead24eb89..463882ca9fa 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -8,15 +8,17 @@ 3. Update "Compatibility and Deprecations" if a new OTP version is supported -4. Commit changes above with title "Release vVERSION", generate a new tag, and push it +4. Commit changes above with title "Release vVERSION" and push it -5. Wait until GitHub Actions publish artifacts to the draft release and the CI is green +5. Once GitHub actions completes, generate a new tag, and push it -6. Copy the relevant bits from /CHANGELOG.md to the GitHub release and publish it (link to the announcement if there is one) +6. Wait until GitHub Actions publish artifacts to the draft release -7. Update `_data/elixir-versions.yml` (except for RCs) in `elixir-lang/elixir-lang.github.com` +7. Copy the relevant bits from /CHANGELOG.md to the GitHub release and publish it (link to the announcement if there is one) -## Creating a new vMAJOR.MINOR branch (after first rc) +8. Update `_data/elixir-versions.yml` (except for RCs) in `elixir-lang/elixir-lang.github.com` + +## Creating a new vMAJOR.MINOR branch (before first rc) ### In the new branch diff --git a/SECURITY.md b/SECURITY.md index a5ec4489d62..f575da4c573 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,12 +6,11 @@ Elixir applies bug fixes only to the latest minor branch. Security patches are a Elixir version | Support :------------- | :----------------------------- -1.17 | Development -1.16 | Bug fixes and security patches +1.17 | Bug fixes and security patches +1.16 | Security patches only 1.15 | Security patches only 1.14 | Security patches only 1.13 | Security patches only -1.12 | Security patches only ## Announcements diff --git a/VERSION b/VERSION index ee8855caa4a..b9a05a6dc1d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.17.0-dev +1.17.3 diff --git a/bin/elixir b/bin/elixir index 87b7df20130..c45d2727527 100755 --- a/bin/elixir +++ b/bin/elixir @@ -1,7 +1,7 @@ #!/bin/sh set -e -ELIXIR_VERSION=1.17.0-dev +ELIXIR_VERSION=1.17.3 if [ $# -eq 0 ] || { [ $# -eq 1 ] && { [ "$1" = "--help" ] || [ "$1" = "-h" ]; }; }; then cat <&2 diff --git a/bin/elixir.bat b/bin/elixir.bat index 13580fa5e5c..0fd800981ac 100644 --- a/bin/elixir.bat +++ b/bin/elixir.bat @@ -1,6 +1,6 @@ @echo off -set ELIXIR_VERSION=1.17.0-dev +set ELIXIR_VERSION=1.17.3 if ""%1""=="""" if ""%2""=="""" goto documentation if /I ""%1""==""--help"" if ""%2""=="""" goto documentation diff --git a/lib/eex/lib/eex/compiler.ex b/lib/eex/lib/eex/compiler.ex index 63910400c6c..0b9eb4a069e 100644 --- a/lib/eex/lib/eex/compiler.ex +++ b/lib/eex/lib/eex/compiler.ex @@ -340,13 +340,6 @@ defmodule EEx.Compiler do scope, state ) do - if mark == ~c"" do - message = - "the contents of this expression won't be output unless the EEx block starts with \"<%=\"" - - :elixir_errors.erl_warn({meta.line, meta.column}, state.file, message) - end - {rest, line, contents} = look_ahead_middle(rest, meta.line, chars) || {rest, meta.line, chars} start_line = meta.line start_column = column(meta.column, mark) @@ -359,6 +352,13 @@ defmodule EEx.Compiler do %{state | quoted: [], line: line} ) + if mark == ~c"" and not match?({:=, _, [_, _]}, contents) do + message = + "the contents of this expression won't be output unless the EEx block starts with \"<%=\"" + + :elixir_errors.erl_warn({meta.line, meta.column}, state.file, message) + end + buffer = state.engine.handle_expr(buffer, IO.chardata_to_string(mark), contents) generate_buffer(rest, buffer, scope, state) end diff --git a/lib/eex/test/eex/smart_engine_test.exs b/lib/eex/test/eex/smart_engine_test.exs index f7a18323e50..de0ee6a1953 100644 --- a/lib/eex/test/eex/smart_engine_test.exs +++ b/lib/eex/test/eex/smart_engine_test.exs @@ -43,15 +43,6 @@ defmodule EEx.SmartEngineTest do assert_received :found end - test "error with unused \"do\" block without \"<%=\" modifier" do - stderr = - ExUnit.CaptureIO.capture_io(:stderr, fn -> - assert_eval("", "<% if true do %>I'm invisible!<% end %>", assigns: %{}) - end) - - assert stderr =~ "the contents of this expression won't be output" - end - defp assert_eval(expected, actual, binding \\ []) do result = EEx.eval_string(actual, binding, file: __ENV__.file, engine: EEx.SmartEngine) assert result == expected diff --git a/lib/eex/test/eex_test.exs b/lib/eex/test/eex_test.exs index 8436bd65b3b..021e3f04caa 100644 --- a/lib/eex/test/eex_test.exs +++ b/lib/eex/test/eex_test.exs @@ -543,6 +543,16 @@ defmodule EExTest do ~s[unexpected beginning of EEx tag \"<%=\" on \"<%= end %>\"] end + test "unused \"do\" block without \"<%=\" modifier" do + assert ExUnit.CaptureIO.capture_io(:stderr, fn -> + EEx.compile_string("<% if true do %>I'm invisible!<% end %>") + end) =~ "the contents of this expression won't be output" + + # These are fine though + EEx.compile_string("<% foo = fn -> %>Hello<% end %>") + EEx.compile_string("<% foo = if true do %>Hello<% end %>") + end + test "from tokenizer" do warning = ExUnit.CaptureIO.capture_io(:stderr, fn -> diff --git a/lib/elixir/lib/calendar/date.ex b/lib/elixir/lib/calendar/date.ex index a14d561f75f..b9bc63e7266 100644 --- a/lib/elixir/lib/calendar/date.ex +++ b/lib/elixir/lib/calendar/date.ex @@ -764,8 +764,9 @@ defmodule Date do When using the default ISO calendar, durations are collapsed and applied in the order of months and then days: - - when shifting by 1 year and 2 months the date is actually shifted by 14 months - - when shifting by 2 weeks and 3 days the date is shifted by 17 days + + * when shifting by 1 year and 2 months the date is actually shifted by 14 months + * when shifting by 2 weeks and 3 days the date is shifted by 17 days When shifting by month, days are rounded down to the nearest valid date. diff --git a/lib/elixir/lib/calendar/datetime.ex b/lib/elixir/lib/calendar/datetime.ex index 3e5ae413a0e..c6cf9c18841 100644 --- a/lib/elixir/lib/calendar/datetime.ex +++ b/lib/elixir/lib/calendar/datetime.ex @@ -1708,8 +1708,9 @@ defmodule DateTime do When using the default ISO calendar, durations are collapsed and applied in the order of months, then seconds and microseconds: - - when shifting by 1 year and 2 months the date is actually shifted by 14 months - - weeks, days and smaller units are collapsed into seconds and microseconds + + * when shifting by 1 year and 2 months the date is actually shifted by 14 months + * weeks, days and smaller units are collapsed into seconds and microseconds When shifting by month, days are rounded down to the nearest valid date. diff --git a/lib/elixir/lib/calendar/duration.ex b/lib/elixir/lib/calendar/duration.ex index 95cf9ca3937..1690332f148 100644 --- a/lib/elixir/lib/calendar/duration.ex +++ b/lib/elixir/lib/calendar/duration.ex @@ -26,7 +26,7 @@ defmodule Duration do iex> Date.shift(~D[2016-01-03], Duration.new!(month: 2)) ~D[2016-03-03] - It is important to notice that shifting is not an arithmetic operation. + It is important to note that shifting is not an arithmetic operation. For example, adding `date + 1 month + 1 month` does not yield the same result as `date + 2 months`. Let's see an example: @@ -46,9 +46,19 @@ defmodule Duration do * larger units (such as years and months) are applied before smaller ones (such as weeks, hours, days, and so on) + * units are collapsed into months (`:year` and `:month`), + seconds (`:week`, `:day`, `:hour`, `:minute`, `:second`) + and microseconds (`:microsecond`) before they are applied + + * 1 year is equivalent to 12 months, 1 week is equivalent to 7 days. + Therefore, 4 weeks _are not_ equivalent to 1 month + * in case of non-existing dates, the results are rounded down to the nearest valid date + As the `shift/2` functions are calendar aware, they are guaranteed to return + valid date/times, considering leap years as well as DST in applicable time zones. + ## Intervals Durations in Elixir can be combined with stream operations to build intervals. @@ -124,6 +134,8 @@ defmodule Duration do """ @type duration :: t | [unit_pair] + @microseconds_per_second 1_000_000 + @doc """ Creates a new `Duration` struct from given `unit_pairs`. @@ -181,9 +193,9 @@ defmodule Duration do ## Examples - iex> Duration.add(%Duration{week: 2, day: 1}, %Duration{day: 2}) + iex> Duration.add(Duration.new!(week: 2, day: 1), Duration.new!(day: 2)) %Duration{week: 2, day: 3} - iex> Duration.add(%Duration{microsecond: {400, 3}}, %Duration{microsecond: {600, 6}}) + iex> Duration.add(Duration.new!(microsecond: {400, 3}), Duration.new!(microsecond: {600, 6})) %Duration{microsecond: {1000, 6}} """ @@ -211,9 +223,9 @@ defmodule Duration do ## Examples - iex> Duration.subtract(%Duration{week: 2, day: 1}, %Duration{day: 2}) + iex> Duration.subtract(Duration.new!(week: 2, day: 1), Duration.new!(day: 2)) %Duration{week: 2, day: -1} - iex> Duration.subtract(%Duration{microsecond: {400, 6}}, %Duration{microsecond: {600, 3}}) + iex> Duration.subtract(Duration.new!(microsecond: {400, 6}), Duration.new!(microsecond: {600, 3})) %Duration{microsecond: {-200, 6}} """ @@ -239,9 +251,9 @@ defmodule Duration do ## Examples - iex> Duration.multiply(%Duration{day: 1, minute: 15, second: -10}, 3) + iex> Duration.multiply(Duration.new!(day: 1, minute: 15, second: -10), 3) %Duration{day: 3, minute: 45, second: -30} - iex> Duration.multiply(%Duration{microsecond: {200, 4}}, 3) + iex> Duration.multiply(Duration.new!(microsecond: {200, 4}), 3) %Duration{microsecond: {600, 4}} """ @@ -264,9 +276,9 @@ defmodule Duration do ## Examples - iex> Duration.negate(%Duration{day: 1, minute: 15, second: -10}) + iex> Duration.negate(Duration.new!(day: 1, minute: 15, second: -10)) %Duration{day: -1, minute: -15, second: 10} - iex> Duration.negate(%Duration{microsecond: {500000, 4}}) + iex> Duration.negate(Duration.new!(microsecond: {500000, 4})) %Duration{microsecond: {-500000, 4}} """ @@ -283,4 +295,142 @@ defmodule Duration do microsecond: {-ms, p} } end + + @doc """ + Parses an [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) formatted duration string to a `Duration` struct. + + Duration strings, as well as individual units, may be prefixed with plus/minus signs so that: + + - `-PT6H3M` parses as `%Duration{hour: -6, minute: -3}` + - `-PT6H-3M` parses as `%Duration{hour: -6, minute: 3}` + - `+PT6H3M` parses as `%Duration{hour: 6, minute: 3}` + - `+PT6H-3M` parses as `%Duration{hour: 6, minute: -3}` + + Duration designators must be provided in order of magnitude: `P[n]Y[n]M[n]W[n]DT[n]H[n]M[n]S`. + + Only seconds may be specified with a decimal fraction, using either a comma or a full stop: `P1DT4,5S`. + + ## Examples + + iex> Duration.from_iso8601("P1Y2M3DT4H5M6S") + {:ok, %Duration{year: 1, month: 2, day: 3, hour: 4, minute: 5, second: 6}} + iex> Duration.from_iso8601("P3Y-2MT3H") + {:ok, %Duration{year: 3, month: -2, hour: 3}} + iex> Duration.from_iso8601("-PT10H-30M") + {:ok, %Duration{hour: -10, minute: 30}} + iex> Duration.from_iso8601("PT4.650S") + {:ok, %Duration{second: 4, microsecond: {650000, 3}}} + + """ + @spec from_iso8601(String.t()) :: {:ok, t} | {:error, atom} + def from_iso8601(string) when is_binary(string) do + case Calendar.ISO.parse_duration(string) do + {:ok, duration} -> + {:ok, new!(duration)} + + error -> + error + end + end + + @doc """ + Same as `from_iso8601/1` but raises an `ArgumentError`. + + ## Examples + + iex> Duration.from_iso8601!("P1Y2M3DT4H5M6S") + %Duration{year: 1, month: 2, day: 3, hour: 4, minute: 5, second: 6} + iex> Duration.from_iso8601!("P10D") + %Duration{day: 10} + + """ + @spec from_iso8601!(String.t()) :: t + def from_iso8601!(string) when is_binary(string) do + case from_iso8601(string) do + {:ok, duration} -> + duration + + {:error, reason} -> + raise ArgumentError, ~s/failed to parse duration "#{string}". reason: #{inspect(reason)}/ + end + end + + @doc """ + Converts the given `duration` to an [ISO 8601-2:2019](https://en.wikipedia.org/wiki/ISO_8601) formatted string. + + Note this function implements the *extension* of ISO 8601:2019. This extensions allows weeks to + appear between months and days: `P3M3W3D`, making it fully compatible with any `Duration` struct. + + ## Examples + + iex> Duration.to_iso8601(Duration.new!(year: 3)) + "P3Y" + iex> Duration.to_iso8601(Duration.new!(day: 40, hour: 12, minute: 42, second: 12)) + "P40DT12H42M12S" + iex> Duration.to_iso8601(Duration.new!(second: 30)) + "PT30S" + + iex> Duration.to_iso8601(Duration.new!([])) + "PT0S" + + iex> Duration.to_iso8601(Duration.new!(second: 1, microsecond: {2_200, 3})) + "PT1.002S" + iex> Duration.to_iso8601(Duration.new!(second: 1, microsecond: {-1_200_000, 4})) + "PT-0.2000S" + """ + + @spec to_iso8601(t) :: String.t() + def to_iso8601(%Duration{} = duration) do + case {to_iso8601_duration_date(duration), to_iso8601_duration_time(duration)} do + {[], []} -> "PT0S" + {date, time} -> IO.iodata_to_binary([?P, date, time]) + end + end + + defp to_iso8601_duration_date(%{year: 0, month: 0, week: 0, day: 0}) do + [] + end + + defp to_iso8601_duration_date(%{year: year, month: month, week: week, day: day}) do + [pair(year, ?Y), pair(month, ?M), pair(week, ?W), pair(day, ?D)] + end + + defp to_iso8601_duration_time(%{hour: 0, minute: 0, second: 0, microsecond: {0, _}}) do + [] + end + + defp to_iso8601_duration_time(%{hour: hour, minute: minute} = d) do + [?T, pair(hour, ?H), pair(minute, ?M), second_component(d)] + end + + defp second_component(%{second: 0, microsecond: {0, _}}) do + [] + end + + defp second_component(%{second: 0, microsecond: {_, 0}}) do + ~c"0S" + end + + defp second_component(%{second: second, microsecond: {_, 0}}) do + [Integer.to_string(second), ?S] + end + + defp second_component(%{second: second, microsecond: {ms, p}}) do + total_ms = second * @microseconds_per_second + ms + second = total_ms |> div(@microseconds_per_second) |> abs() + ms = total_ms |> rem(@microseconds_per_second) |> abs() + sign = if total_ms < 0, do: ?-, else: [] + + [ + sign, + Integer.to_string(second), + ?., + ms |> Integer.to_string() |> String.pad_leading(6, "0") |> binary_part(0, p), + ?S + ] + end + + @compile {:inline, pair: 2} + defp pair(0, _key), do: [] + defp pair(num, key), do: [Integer.to_string(num), key] end diff --git a/lib/elixir/lib/calendar/iso.ex b/lib/elixir/lib/calendar/iso.ex index 18072edf0d0..d016bf8113b 100644 --- a/lib/elixir/lib/calendar/iso.ex +++ b/lib/elixir/lib/calendar/iso.ex @@ -18,7 +18,8 @@ defmodule Calendar.ISO do The standard library supports a minimal set of possible ISO 8601 features. Specifically, the parser only supports calendar dates and does not support - ordinal and week formats. + ordinal and week formats. Additionally, it supports parsing ISO 8601 + formatted durations, including negative time units and fractional seconds. By default Elixir only parses extended-formatted date/times. You can opt-in to parse basic-formatted date/times. @@ -29,7 +30,7 @@ defmodule Calendar.ISO do Elixir does not support reduced accuracy formats (for example, a date without the day component) nor decimal precisions in the lowest component (such as - `10:01:25,5`). No functions exist to parse ISO 8601 durations or time intervals. + `10:01:25,5`). #### Examples @@ -663,6 +664,87 @@ defmodule Calendar.ISO do end end + @doc """ + Parses an ISO 8601 formatted duration string to a list of `Duration` compabitble unit pairs. + + See `Duration.from_iso8601/1`. + """ + @doc since: "1.17.0" + @spec parse_duration(String.t()) :: {:ok, [Duration.unit_pair()]} | {:error, atom} + def parse_duration("P" <> string) when byte_size(string) > 0 do + parse_duration_date(string, [], year: ?Y, month: ?M, week: ?W, day: ?D) + end + + def parse_duration("+P" <> string) when byte_size(string) > 0 do + parse_duration_date(string, [], year: ?Y, month: ?M, week: ?W, day: ?D) + end + + def parse_duration("-P" <> string) when byte_size(string) > 0 do + with {:ok, fields} <- parse_duration_date(string, [], year: ?Y, month: ?M, week: ?W, day: ?D) do + {:ok, + Enum.map(fields, fn + {:microsecond, {value, precision}} -> {:microsecond, {-value, precision}} + {unit, value} -> {unit, -value} + end)} + end + end + + def parse_duration(_) do + {:error, :invalid_duration} + end + + defp parse_duration_date("", acc, _allowed), do: {:ok, acc} + + defp parse_duration_date("T" <> string, acc, _allowed) when byte_size(string) > 0 do + parse_duration_time(string, acc, hour: ?H, minute: ?M, second: ?S) + end + + defp parse_duration_date(string, acc, allowed) do + with {integer, <>} <- Integer.parse(string), + {key, allowed} <- find_unit(allowed, next) do + parse_duration_date(rest, [{key, integer} | acc], allowed) + else + _ -> {:error, :invalid_date_component} + end + end + + defp parse_duration_time("", acc, _allowed), do: {:ok, acc} + + defp parse_duration_time(string, acc, allowed) do + case Integer.parse(string) do + {second, <> = rest} when delimiter in [?., ?,] -> + case parse_microsecond(rest) do + {{ms, precision}, "S"} -> + ms = + case string do + "-" <> _ -> + -ms + + _ -> + ms + end + + {:ok, [second: second, microsecond: {ms, precision}] ++ acc} + + _ -> + {:error, :invalid_time_component} + end + + {integer, <>} -> + case find_unit(allowed, next) do + {key, allowed} -> parse_duration_time(rest, [{key, integer} | acc], allowed) + false -> {:error, :invalid_time_component} + end + + _ -> + {:error, :invalid_time_component} + end + end + + defp find_unit([{key, unit} | rest], unit), do: {key, rest} + defp find_unit([_ | rest], unit), do: find_unit(rest, unit) + defp find_unit([], _unit), do: false + @doc """ Returns the `t:Calendar.iso_days/0` format of the specified date. diff --git a/lib/elixir/lib/calendar/naive_datetime.ex b/lib/elixir/lib/calendar/naive_datetime.ex index 6df406a8cbb..059b5bbc7e2 100644 --- a/lib/elixir/lib/calendar/naive_datetime.ex +++ b/lib/elixir/lib/calendar/naive_datetime.ex @@ -578,8 +578,9 @@ defmodule NaiveDateTime do When using the default ISO calendar, durations are collapsed and applied in the order of months, then seconds and microseconds: - - when shifting by 1 year and 2 months the date is actually shifted by 14 months - - weeks, days and smaller units are collapsed into seconds and microseconds + + * when shifting by 1 year and 2 months the date is actually shifted by 14 months + * weeks, days and smaller units are collapsed into seconds and microseconds When shifting by month, days are rounded down to the nearest valid date. diff --git a/lib/elixir/lib/code/normalizer.ex b/lib/elixir/lib/code/normalizer.ex index f86ca831424..2d38844f29e 100644 --- a/lib/elixir/lib/code/normalizer.ex +++ b/lib/elixir/lib/code/normalizer.ex @@ -174,16 +174,17 @@ defmodule Code.Normalizer do # Sigils defp do_normalize({sigil, meta, [{:<<>>, _, args} = string, modifiers]} = quoted, state) - when is_list(args) and is_atom(sigil) do - case Atom.to_string(sigil) do - "sigil_" <> _ -> - meta = - meta - |> patch_meta_line(state.parent_meta) - |> Keyword.put_new(:delimiter, "\"") - - {sigil, meta, [do_normalize(string, %{state | parent_meta: meta}), modifiers]} + when is_atom(sigil) and is_list(args) and is_list(modifiers) do + with "sigil_" <> _ <- Atom.to_string(sigil), + true <- binary_interpolated?(args), + true <- List.ascii_printable?(modifiers) do + meta = + meta + |> patch_meta_line(state.parent_meta) + |> Keyword.put_new(:delimiter, "\"") + {sigil, meta, [do_normalize(string, %{state | parent_meta: meta}), modifiers]} + else _ -> normalize_call(quoted, state) end diff --git a/lib/elixir/lib/exception.ex b/lib/elixir/lib/exception.ex index 9641b405cd6..c24d8904ad8 100644 --- a/lib/elixir/lib/exception.ex +++ b/lib/elixir/lib/exception.ex @@ -1212,7 +1212,7 @@ defmodule SyntaxError do }) when not is_nil(snippet) and not is_nil(column) do snippet = - :elixir_errors.format_snippet({line, column}, file, description, snippet, :error, [], nil) + :elixir_errors.format_snippet(:error, {line, column}, file, description, snippet, %{}) format_message(file, line, column, snippet) end @@ -1225,7 +1225,7 @@ defmodule SyntaxError do description: description }) do snippet = - :elixir_errors.format_snippet({line, column}, file, description, nil, :error, [], nil) + :elixir_errors.format_snippet(:error, {line, column}, file, description, nil, %{}) padded = " " <> String.replace(snippet, "\n", "\n ") format_message(file, line, column, padded) @@ -1316,7 +1316,7 @@ defmodule TokenMissingError do description: description }) do snippet = - :elixir_errors.format_snippet({line, column}, file, description, snippet, :error, [], nil) + :elixir_errors.format_snippet(:error, {line, column}, file, description, snippet, %{}) format_message(file, line, column, snippet) end diff --git a/lib/elixir/lib/inspect/algebra.ex b/lib/elixir/lib/inspect/algebra.ex index 5e213b37798..189a4cbfd01 100644 --- a/lib/elixir/lib/inspect/algebra.ex +++ b/lib/elixir/lib/inspect/algebra.ex @@ -588,6 +588,21 @@ defmodule Inspect.Algebra do doc_cons(doc1, doc2) end + @doc ~S""" + Disable any rendering limit while rendering the given document. + + ## Examples + + iex> doc = Inspect.Algebra.glue("hello", "world") |> Inspect.Algebra.group() + iex> Inspect.Algebra.format(doc, 10) + ["hello", "\n", "world"] + iex> doc = Inspect.Algebra.no_limit(doc) + iex> Inspect.Algebra.format(doc, 10) + ["hello", " ", "world"] + + """ + @doc since: "1.14.0" + @spec no_limit(t) :: t def no_limit(doc) do doc_limit(doc, :infinity) end diff --git a/lib/elixir/lib/kernel.ex b/lib/elixir/lib/kernel.ex index e96d25cfb2c..295181be6ea 100644 --- a/lib/elixir/lib/kernel.ex +++ b/lib/elixir/lib/kernel.ex @@ -6197,9 +6197,8 @@ defmodule Kernel do """ @doc since: "1.17.0" - @spec to_timeout([component, ...] | timeout() | Duration.t()) :: timeout() - when component: [{unit, non_neg_integer()}, ...], - unit: :week | :day | :hour | :minute | :second | :millisecond + @spec to_timeout([{unit, non_neg_integer()}] | timeout() | Duration.t()) :: timeout() + when unit: :week | :day | :hour | :minute | :second | :millisecond def to_timeout(duration) def to_timeout(:infinity), do: :infinity diff --git a/lib/elixir/lib/keyword.ex b/lib/elixir/lib/keyword.ex index 79f0bd5add8..466511b228a 100644 --- a/lib/elixir/lib/keyword.ex +++ b/lib/elixir/lib/keyword.ex @@ -960,6 +960,36 @@ defmodule Keyword do :lists.sort(left) === :lists.sort(right) end + @doc """ + Intersects two keyword lists, returning a keyword with the common keys. + + By default, it returns the values of the intersected keys in `keyword2`. + The keys are returned in the order found in `keyword1`. + + ## Examples + + iex> Keyword.intersect([a: 1, b: 2], [b: "b", c: "c"]) + [b: "b"] + + iex> Keyword.intersect([a: 1, b: 2], [b: 2, c: 3], fn _k, v1, v2 -> + ...> v1 + v2 + ...> end) + [b: 4] + + """ + @doc since: "1.17.0" + @spec intersect(keyword, keyword, (key, value, value -> value)) :: keyword + def intersect(keyword1, keyword2, fun \\ fn _key, _v1, v2 -> v2 end) + + def intersect([{k, v1} | keyword1], keyword2, fun) do + case :lists.keyfind(k, 1, keyword2) do + {_, v2} -> [{k, fun.(k, v1, v2)} | intersect(keyword1, keyword2, fun)] + false -> intersect(keyword1, keyword2, fun) + end + end + + def intersect([], _keyword2, _fun), do: [] + @doc """ Merges two keyword lists into one. diff --git a/lib/elixir/lib/macro.ex b/lib/elixir/lib/macro.ex index e6ded3f4eb7..964d74d5ffd 100644 --- a/lib/elixir/lib/macro.ex +++ b/lib/elixir/lib/macro.ex @@ -495,10 +495,8 @@ defmodule Macro do """ @doc since: "1.11.3" @spec generate_unique_arguments(0, context :: atom) :: [] - @spec generate_unique_arguments(pos_integer, context) :: [ - {atom, [counter: integer], context}, - ... - ] + @spec generate_unique_arguments(pos_integer, context) :: + [{atom, [counter: integer], context}, ...] when context: atom def generate_unique_arguments(amount, context), do: generate_arguments(amount, context, &unique_var/2) @@ -1834,13 +1832,13 @@ defmodule Macro do defp do_expand_once({:__DIR__, _, atom}, env) when is_atom(atom), do: {:filename.dirname(env.file), true} - defp do_expand_once({:__ENV__, _, atom}, env) when is_atom(atom) do + defp do_expand_once({:__ENV__, _, atom}, env) when is_atom(atom) and env.context != :match do env = update_in(env.versioned_vars, &maybe_escape_map/1) {maybe_escape_map(env), true} end defp do_expand_once({{:., _, [{:__ENV__, _, atom}, field]}, _, []} = original, env) - when is_atom(atom) and is_atom(field) do + when is_atom(atom) and is_atom(field) and env.context != :match do if Map.has_key?(env, field) do {maybe_escape_map(Map.get(env, field)), true} else diff --git a/lib/elixir/lib/macro/env.ex b/lib/elixir/lib/macro/env.ex index 2afb266dcea..ba00d6922c3 100644 --- a/lib/elixir/lib/macro/env.ex +++ b/lib/elixir/lib/macro/env.ex @@ -349,6 +349,11 @@ defmodule Macro.Env do * #{trace_option} + * `:info_callback` - a function to use instead of `c:Module.__info__/1`. + The function will be invoked with `:functions` or `:macros` argument. + It has to return a list of `{function, arity}` key value pairs. + If it fails, it defaults to using module metadata based on `module_info/1`. + ## Examples iex> env = __ENV__ @@ -367,15 +372,27 @@ defmodule Macro.Env do iex> Macro.Env.lookup_import(env, {:is_odd, 1}) [{:macro, Integer}] + ## Info callback override + + iex> env = __ENV__ + iex> Macro.Env.lookup_import(env, {:flatten, 1}) + [] + iex> {:ok, env} = Macro.Env.define_import(env, [line: 10], SomeModule, [info_callback: fn :functions -> [{:flatten, 1}]; :macros -> [{:some, 2}]; end]) + iex> Macro.Env.lookup_import(env, {:flatten, 1}) + [{:function, SomeModule}] + iex> Macro.Env.lookup_import(env, {:some, 2}) + [{:macro, SomeModule}] + """ @doc since: "1.17.0" - @spec define_import(t, Macro.metadata(), module) :: {:ok, t} | {:error, String.t()} + @spec define_import(t, Macro.metadata(), module, keyword) :: {:ok, t} | {:error, String.t()} def define_import(env, meta, module, opts \\ []) when is_list(meta) and is_atom(module) and is_list(opts) do {trace, opts} = Keyword.pop(opts, :trace, true) {warnings, opts} = Keyword.pop(opts, :emit_warnings, true) + {info_callback, opts} = Keyword.pop(opts, :info_callback, &module.__info__/1) - result = :elixir_import.import(meta, module, opts, env, warnings, trace) + result = :elixir_import.import(meta, module, opts, env, warnings, trace, info_callback) maybe_define_error(result, :elixir_import) end @@ -583,7 +600,7 @@ defmodule Macro.Env do :elixir_dispatch.check_deprecated(:macro, meta, receiver, name, arity, env) end - quoted = expander.(args, :elixir_env.env_to_ex(env)) + quoted = expander.(args, env) next = :elixir_module.next_counter(env.module) :elixir_quote.linify_with_context_counter(expansion_meta, {receiver, next}, quoted) end diff --git a/lib/elixir/lib/module/behaviour.ex b/lib/elixir/lib/module/behaviour.ex index 2cc857d132e..e37f43852e4 100644 --- a/lib/elixir/lib/module/behaviour.ex +++ b/lib/elixir/lib/module/behaviour.ex @@ -284,7 +284,11 @@ defmodule Module.Behaviour do end end - def format_warning({:undefined_behaviour, module, behaviour}) do + def format_diagnostic(warning) do + %{message: IO.iodata_to_binary(format_warning(warning))} + end + + defp format_warning({:undefined_behaviour, module, behaviour}) do [ "@behaviour ", inspect(behaviour), @@ -294,12 +298,12 @@ defmodule Module.Behaviour do ] end - def format_warning({:module_does_not_define_behaviour, module, behaviour}) do + defp format_warning({:module_does_not_define_behaviour, module, behaviour}) do ["module ", inspect(behaviour), " is not a behaviour (in module ", inspect(module), ")"] end - def format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback}) - when conflict == behaviour do + defp format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback}) + when conflict == behaviour do [ "the behaviour ", inspect(behaviour), @@ -311,7 +315,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback}) do + defp format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback}) do [ "conflicting behaviours found. Callback ", format_definition(kind, callback), @@ -325,7 +329,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:missing_callback, module, callback, kind, behaviour}) do + defp format_warning({:missing_callback, module, callback, kind, behaviour}) do [ format_callback(callback, kind, behaviour), " is not implemented (in module ", @@ -334,7 +338,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:callback_mismatch, module, callback, kind, wrong_kind, behaviour}) do + defp format_warning({:callback_mismatch, module, callback, kind, wrong_kind, behaviour}) do [ format_callback(callback, kind, behaviour), " was implemented as \"", @@ -347,14 +351,14 @@ defmodule Module.Behaviour do ] end - def format_warning({:private_function, callback, kind}) do + defp format_warning({:private_function, callback, kind}) do [ format_definition(kind, callback), " is private, @impl attribute is always discarded for private functions/macros" ] end - def format_warning({:no_behaviours, callback, kind, value}) do + defp format_warning({:no_behaviours, callback, kind, value}) do [ "got \"@impl ", inspect(value), @@ -364,7 +368,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:impl_not_defined, callback, kind, {_fa, behaviour}}) do + defp format_warning({:impl_not_defined, callback, kind, {_fa, behaviour}}) do [ "got \"@impl false\" for ", format_definition(kind, callback), @@ -373,7 +377,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:impl_defined, callback, kind, callbacks}) do + defp format_warning({:impl_defined, callback, kind, callbacks}) do [ "got \"@impl true\" for ", format_definition(kind, callback), @@ -382,7 +386,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:behaviour_not_declared, callback, kind, behaviour}) do + defp format_warning({:behaviour_not_declared, callback, kind, behaviour}) do [ "got \"@impl ", inspect(behaviour), @@ -392,7 +396,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:behaviour_not_defined, callback, kind, behaviour, callbacks}) do + defp format_warning({:behaviour_not_defined, callback, kind, behaviour, callbacks}) do [ "got \"@impl ", inspect(behaviour), @@ -403,7 +407,7 @@ defmodule Module.Behaviour do ] end - def format_warning({:missing_impl, callback, kind, behaviour}) do + defp format_warning({:missing_impl, callback, kind, behaviour}) do [ "module attribute @impl was not set for ", format_definition(kind, callback), diff --git a/lib/elixir/lib/module/parallel_checker.ex b/lib/elixir/lib/module/parallel_checker.ex index 6c063237693..02d8dc32bd0 100644 --- a/lib/elixir/lib/module/parallel_checker.ex +++ b/lib/elixir/lib/module/parallel_checker.ex @@ -259,7 +259,7 @@ defmodule Module.ParallelChecker do definitions ) - warnings = + diagnostics = module |> Module.Types.warnings(file, definitions, no_warn_undefined, cache) |> Kernel.++(behaviour_warnings) @@ -270,7 +270,7 @@ defmodule Module.ParallelChecker do |> Map.get(:after_verify, []) |> Enum.each(fn {verify_mod, verify_fun} -> apply(verify_mod, verify_fun, [module]) end) - warnings + diagnostics end defp extract_no_warn_undefined(compile_opts) do @@ -291,42 +291,58 @@ defmodule Module.ParallelChecker do ## Warning helpers defp group_warnings(warnings) do - warnings - |> Enum.reduce(%{}, fn {module, warning, location}, acc -> - locations = MapSet.new([location]) - Map.update(acc, {module, warning}, locations, &MapSet.put(&1, location)) - end) - |> Enum.map(fn {{module, warning}, locations} -> {module, warning, Enum.sort(locations)} end) - |> Enum.sort() + {ungrouped, grouped} = + Enum.reduce(warnings, {[], %{}}, fn {module, warning, location}, {ungrouped, grouped} -> + %{message: _} = diagnostic = module.format_diagnostic(warning) + + if Map.get(diagnostic, :group, false) do + locations = MapSet.new([location]) + + grouped = + Map.update(grouped, warning, {locations, diagnostic}, fn + {locations, diagnostic} -> {MapSet.put(locations, location), diagnostic} + end) + + {ungrouped, grouped} + else + {[{[location], diagnostic} | ungrouped], grouped} + end + end) + + grouped = + Enum.map(grouped, fn {_warning, {locations, diagnostic}} -> + {Enum.sort(locations), diagnostic} + end) + + Enum.sort(ungrouped ++ grouped) end defp emit_warnings(warnings, log?) do - Enum.flat_map(warnings, fn {module, warning, locations} -> - message = module.format_warning(warning) - diagnostics = Enum.map(locations, &to_diagnostic(message, &1)) - log? and print_warning(message, diagnostics) + Enum.flat_map(warnings, fn {locations, diagnostic} -> + diagnostics = Enum.map(locations, &to_diagnostic(diagnostic, &1)) + log? and print_diagnostics(diagnostics) diagnostics end) end - defp print_warning(message, [diagnostic]) do - :elixir_errors.print_warning(message, diagnostic) + defp print_diagnostics([diagnostic]) do + :elixir_errors.print_diagnostic(diagnostic, true) end - defp print_warning(message, grouped_warnings) do - :elixir_errors.print_warning_group(message, grouped_warnings) + defp print_diagnostics(diagnostics) do + :elixir_errors.print_diagnostics(diagnostics) end - defp to_diagnostic(message, {file, position, mfa}) when is_list(position) do + defp to_diagnostic(diagnostic, {file, position, mfa}) when is_list(position) do %{ severity: :warning, source: file, file: file, position: position_to_tuple(position), - message: IO.iodata_to_binary(message), stacktrace: [to_stacktrace(file, position, mfa)], span: nil } + |> Map.merge(diagnostic) end defp position_to_tuple(position) do diff --git a/lib/elixir/lib/module/types/descr.ex b/lib/elixir/lib/module/types/descr.ex index f2cc19b5f3f..b046507a5c1 100644 --- a/lib/elixir/lib/module/types/descr.ex +++ b/lib/elixir/lib/module/types/descr.ex @@ -34,18 +34,17 @@ defmodule Module.Types.Descr do @atom_top {:negation, :sets.new(version: 2)} @map_top [{:open, %{}, []}] @map_empty [{:closed, %{}, []}] - - # Guard helpers - - @term %{bitmap: @bit_top, atom: @atom_top, map: @map_top} @none %{} - @dynamic %{dynamic: @term} # Type definitions - def dynamic(), do: @dynamic - def term(), do: @term + def dynamic(), do: %{dynamic: :term} def none(), do: @none + def term(), do: :term + + defp unfold(:term), do: unfolded_term() + defp unfold(other), do: other + defp unfolded_term, do: %{bitmap: @bit_top, atom: @atom_top, map: @map_top} def atom(as), do: %{atom: atom_new(as)} def atom(), do: %{atom: @atom_top} @@ -84,14 +83,27 @@ defmodule Module.Types.Descr do @term_or_optional %{bitmap: @bit_top ||| @bit_optional, atom: @atom_top, map: @map_top} def not_set(), do: @not_set - def if_set(type), do: Map.update(type, :bitmap, @bit_optional, &(&1 ||| @bit_optional)) defp term_or_optional(), do: @term_or_optional + def if_set(:term), do: term_or_optional() + def if_set(type), do: Map.update(type, :bitmap, @bit_optional, &(&1 ||| @bit_optional)) + + defguardp is_optional(map) + when is_map(map) and + ((is_map_key(map, :bitmap) and (map.bitmap &&& @bit_optional) != 0) or + (is_map_key(map, :dynamic) and is_map(map.dynamic) and + is_map_key(map.dynamic, :bitmap) and + (map.dynamic.bitmap &&& @bit_optional) != 0)) + + defguardp is_optional_static(map) + when is_map(map) and is_map_key(map, :bitmap) and (map.bitmap &&& @bit_optional) != 0 + ## Set operations - def term_type?(@term), do: true - def term_type?(descr), do: subtype_static(@term, Map.delete(descr, :dynamic)) + def term_type?(:term), do: true + def term_type?(descr), do: subtype_static(unfolded_term(), Map.delete(descr, :dynamic)) + def gradual?(:term), do: false def gradual?(descr), do: is_map_key(descr, :dynamic) @doc """ @@ -102,14 +114,19 @@ defmodule Module.Types.Descr do def dynamic(descr) do case descr do %{dynamic: dynamic} -> %{dynamic: dynamic} - %{} -> %{dynamic: descr} + _ -> %{dynamic: descr} end end @doc """ Computes the union of two descrs. """ - def union(%{} = left, %{} = right) do + def union(:term, other) when not is_optional(other), do: :term + def union(other, :term) when not is_optional(other), do: :term + + def union(left, right) do + left = unfold(left) + right = unfold(right) is_gradual_left = gradual?(left) is_gradual_right = gradual?(right) @@ -136,7 +153,12 @@ defmodule Module.Types.Descr do @doc """ Computes the intersection of two descrs. """ - def intersection(%{} = left, %{} = right) do + def intersection(:term, other) when not is_optional(other), do: other + def intersection(other, :term) when not is_optional(other), do: other + + def intersection(left, right) do + left = unfold(left) + right = unfold(right) is_gradual_left = gradual?(left) is_gradual_right = gradual?(right) @@ -164,14 +186,19 @@ defmodule Module.Types.Descr do @doc """ Computes the difference between two types. """ - def difference(left = %{}, right = %{}) do + def difference(other, :term) when not is_optional(other), do: none() + + def difference(left, right) do + left = unfold(left) + right = unfold(right) + if gradual?(left) or gradual?(right) do {left_dynamic, left_static} = Map.pop(left, :dynamic, left) {right_dynamic, right_static} = Map.pop(right, :dynamic, right) dynamic_part = difference_static(left_dynamic, right_static) if empty?(dynamic_part), - do: @none, + do: none(), else: Map.put(difference_static(left_static, right_dynamic), :dynamic, dynamic_part) else difference_static(left, right) @@ -179,8 +206,10 @@ defmodule Module.Types.Descr do end # For static types, the difference is component-wise. + defp difference_static(left, :term) when not is_optional_static(left), do: none() + defp difference_static(left, right) do - iterator_difference(:maps.next(:maps.iterator(right)), left) + iterator_difference(:maps.next(:maps.iterator(unfold(right))), unfold(left)) end # Returning 0 from the callback is taken as none() for that subtype. @@ -193,7 +222,8 @@ defmodule Module.Types.Descr do @doc """ Compute the negation of a type. """ - def negation(%{} = descr), do: difference(term(), descr) + def negation(:term), do: none() + def negation(%{} = descr), do: difference(unfolded_term(), descr) @doc """ Check if a type is empty. @@ -203,18 +233,26 @@ defmodule Module.Types.Descr do (bitmap, atom) are checked first for speed since, if they are present, the type is non-empty as we normalize then during construction. """ + def empty?(:term), do: false + def empty?(%{} = descr) do - descr = Map.get(descr, :dynamic, descr) + case Map.get(descr, :dynamic, descr) do + :term -> + false - descr == @none or - (not Map.has_key?(descr, :bitmap) and not Map.has_key?(descr, :atom) and - (not Map.has_key?(descr, :map) or map_empty?(descr.map))) + value when value == @none -> + true + + descr -> + not Map.has_key?(descr, :bitmap) and not Map.has_key?(descr, :atom) and + (not Map.has_key?(descr, :map) or map_empty?(descr.map)) + end end @doc """ Converts a descr to its quoted representation. """ - def to_quoted(%{} = descr) do + def to_quoted(descr) do if term_type?(descr) do {:term, [], []} else @@ -260,7 +298,11 @@ defmodule Module.Types.Descr do Because of the dynamic/static invariant in the `descr`, subtyping can be simplified in several cases according to which type is gradual or not. """ - def subtype?(%{} = left, %{} = right) do + def subtype?(left, :term) when not is_optional(left), do: true + + def subtype?(left, right) do + left = unfold(left) + right = unfold(right) is_grad_left = gradual?(left) is_grad_right = gradual?(right) @@ -278,6 +320,7 @@ defmodule Module.Types.Descr do end end + defp subtype_static(same, same), do: true defp subtype_static(left, right), do: empty?(difference_static(left, right)) @doc """ @@ -305,14 +348,18 @@ defmodule Module.Types.Descr do include `dynamic()`, `integer()`, but also `dynamic() and (integer() or atom())`. Incompatible subtypes include `integer() or list()`, `dynamic() and atom()`. """ - def compatible?(input_type, expected_type) do - {input_dynamic, input_static} = Map.pop(input_type, :dynamic, input_type) - expected_dynamic = Map.get(expected_type, :dynamic, expected_type) + def compatible?(left, :term) when not is_optional(left), do: true + + def compatible?(left, right) do + left = unfold(left) + right = unfold(right) + {left_dynamic, left_static} = Map.pop(left, :dynamic, left) + right_dynamic = Map.get(right, :dynamic, right) - if empty?(input_static) do - not empty?(intersection(input_dynamic, expected_dynamic)) + if empty?(left_static) do + not empty?(intersection(left_dynamic, right_dynamic)) else - subtype_static(input_static, expected_dynamic) + subtype_static(left_static, right_dynamic) end end @@ -321,30 +368,38 @@ defmodule Module.Types.Descr do @doc """ Optimized version of `not empty?(intersection(binary(), type))`. """ + def binary_type?(:term), do: true + def binary_type?(%{dynamic: :term}), do: true def binary_type?(%{dynamic: %{bitmap: bitmap}}) when (bitmap &&& @bit_binary) != 0, do: true def binary_type?(%{bitmap: bitmap}) when (bitmap &&& @bit_binary) != 0, do: true - def binary_type?(%{}), do: false + def binary_type?(_), do: false @doc """ Optimized version of `not empty?(intersection(integer(), type))`. """ + def integer_type?(:term), do: true + def integer_type?(%{dynamic: :term}), do: true def integer_type?(%{dynamic: %{bitmap: bitmap}}) when (bitmap &&& @bit_integer) != 0, do: true def integer_type?(%{bitmap: bitmap}) when (bitmap &&& @bit_integer) != 0, do: true - def integer_type?(%{}), do: false + def integer_type?(_), do: false @doc """ Optimized version of `not empty?(intersection(float(), type))`. """ + def float_type?(:term), do: true + def float_type?(%{dynamic: :term}), do: true def float_type?(%{dynamic: %{bitmap: bitmap}}) when (bitmap &&& @bit_float) != 0, do: true def float_type?(%{bitmap: bitmap}) when (bitmap &&& @bit_float) != 0, do: true - def float_type?(%{}), do: false + def float_type?(_), do: false @doc """ Optimized version of `not empty?(intersection(integer() or float(), type))`. """ + def number_type?(:term), do: true + def number_type?(%{dynamic: :term}), do: true def number_type?(%{dynamic: %{bitmap: bitmap}}) when (bitmap &&& @bit_number) != 0, do: true def number_type?(%{bitmap: bitmap}) when (bitmap &&& @bit_number) != 0, do: true - def number_type?(%{}), do: false + def number_type?(_), do: false defp bitmap_union(v1, v2), do: v1 ||| v2 defp bitmap_intersection(v1, v2), do: v1 &&& v2 @@ -370,6 +425,29 @@ defmodule Module.Types.Descr do do: {type, [], []} end + ## Funs + + @doc """ + Checks there is a function type (and only functions) with said arity. + """ + def fun_fetch(:term, _arity), do: :error + + def fun_fetch(%{} = descr, _arity) do + {static_or_dynamic, static} = Map.pop(descr, :dynamic, descr) + + if fun_only?(static) do + case static_or_dynamic do + :term -> :ok + %{bitmap: bitmap} when (bitmap &&& @bit_fun) != 0 -> :ok + %{} -> :error + end + else + :error + end + end + + defp fun_only?(descr), do: empty?(difference(descr, fun())) + ## Atoms # The atom component of a type consists of pairs `{tag, set}` where `set` is a @@ -388,25 +466,18 @@ defmodule Module.Types.Descr do # an empty list of atoms. It is simplified to `0` in set operations, and the key # is removed from the map. - @doc """ - Optimized version of `not empty?(intersection(atom(), type))`. - """ - def atom_type?(%{dynamic: %{atom: _}}), do: true - def atom_type?(%{atom: _}), do: true - def atom_type?(%{}), do: false - @doc """ Optimized version of `not empty?(intersection(atom([atom]), type))`. """ - def atom_type?(%{} = descr, atom) do - {static_or_dynamic, static} = Map.pop(descr, :dynamic, descr) + def atom_type?(:term, _atom), do: true - atom_only?(static) and - case static_or_dynamic do - %{atom: {:union, set}} -> :sets.is_element(atom, set) - %{atom: {:negation, set}} -> not :sets.is_element(atom, set) - %{} -> false - end + def atom_type?(%{} = descr, atom) do + case Map.get(descr, :dynamic, descr) do + :term -> true + %{atom: {:union, set}} -> :sets.is_element(atom, set) + %{atom: {:negation, set}} -> not :sets.is_element(atom, set) + %{} -> false + end end @doc """ @@ -416,11 +487,14 @@ defmodule Module.Types.Descr do `:error` otherwise. Notice `known_set` may be empty in infinite cases, due to negations. """ + def atom_fetch(:term), do: :error + def atom_fetch(%{} = descr) do {static_or_dynamic, static} = Map.pop(descr, :dynamic, descr) if atom_only?(static) do case static_or_dynamic do + :term -> {:infinite, []} %{atom: {:union, set}} -> {:finite, :sets.to_list(set)} %{atom: {:negation, _}} -> {:infinite, []} %{} -> :error @@ -547,19 +621,21 @@ defmodule Module.Types.Descr do # `:dynamic` field is not_set, or it contains a type equal to the static component # (that is, there are no extra dynamic values). - defp dynamic_intersection(left, right) do - inter = symmetrical_intersection(left, right, &intersection/3) - if empty?(inter), do: 0, else: inter - end + defp dynamic_union(:term, other) when not is_optional_static(other), do: :term + defp dynamic_union(other, :term) when not is_optional_static(other), do: :term - defp dynamic_difference(left, right) do - diff = difference_static(left, right) - if empty?(diff), do: 0, else: diff - end + defp dynamic_union(left, right), + do: symmetrical_merge(unfold(left), unfold(right), &union/3) + + defp dynamic_intersection(:term, other) when not is_optional_static(other), do: other + defp dynamic_intersection(other, :term) when not is_optional_static(other), do: other - defp dynamic_union(left, right), do: symmetrical_merge(left, right, &union/3) + defp dynamic_intersection(left, right), + do: symmetrical_intersection(unfold(left), unfold(right), &intersection/3) - defp dynamic_to_quoted(%{} = descr) do + defp dynamic_difference(left, right), do: difference_static(left, right) + + defp dynamic_to_quoted(descr) do cond do term_type?(descr) -> [{:dynamic, [], []}] single = indivisible_bitmap(descr) -> [single] @@ -607,6 +683,10 @@ defmodule Module.Types.Descr do end end + defp map_descr_pairs([{key, :term} | rest], acc, dynamic?) do + map_descr_pairs(rest, [{key, :term} | acc], dynamic?) + end + defp map_descr_pairs([{key, value} | rest], acc, dynamic?) do case :maps.take(:dynamic, value) do :error -> map_descr_pairs(rest, [{key, value} | acc], dynamic?) @@ -618,9 +698,6 @@ defmodule Module.Types.Descr do {acc, dynamic?} end - defp optional?(%{bitmap: bitmap}) when (bitmap &&& @bit_optional) != 0, do: true - defp optional?(_), do: false - defp map_tag_to_type(:open), do: term_or_optional() defp map_tag_to_type(:closed), do: not_set() @@ -635,11 +712,13 @@ defmodule Module.Types.Descr do In static mode, we likely want to raise if `map.field` (or pattern matching?) is called on an optional key. """ + def map_fetch(:term, _key), do: :badmap + def map_fetch(%{} = descr, key) do case :maps.take(:dynamic, descr) do :error -> if is_map_key(descr, :map) and map_only?(descr) do - {static_optional?, static_type} = map_fetch_static(descr, key) |> pop_optional() + {static_optional?, static_type} = map_fetch_static(descr, key) if static_optional? or empty?(static_type) do :badkey @@ -650,13 +729,13 @@ defmodule Module.Types.Descr do :badmap end - {%{map: {:open, fields, []}}, static} when fields == %{} and static == @none -> + {:term, _static} -> {true, dynamic()} {dynamic, static} -> if is_map_key(dynamic, :map) and map_only?(static) do - {dynamic_optional?, dynamic_type} = map_fetch_static(dynamic, key) |> pop_optional() - {static_optional?, static_type} = map_fetch_static(static, key) |> pop_optional() + {dynamic_optional?, dynamic_type} = map_fetch_static(dynamic, key) + {static_optional?, static_type} = map_fetch_static(static, key) if static_optional? or empty?(dynamic_type) do :badkey @@ -673,12 +752,26 @@ defmodule Module.Types.Descr do defp map_fetch_static(descr, key) when is_atom(key) do case descr do - %{map: map} -> Enum.reduce(map_split_on_key(map, key), none(), &union/2) - %{} -> none() + # Optimization: if the key does not exist in the map, + # avoid building if_set/not_set pairs and return the + # popped value directly. + %{map: [{tag, fields, []}]} when not is_map_key(fields, key) -> + case tag do + :open -> {true, term()} + :closed -> {true, none()} + end + + %{map: map} -> + map_split_on_key(map, key) + |> Enum.reduce(none(), &union/2) + |> pop_optional_static() + + %{} -> + {false, none()} end end - defp pop_optional(type) do + defp pop_optional_static(type) do case type do %{bitmap: @bit_optional} -> {true, Map.delete(type, :bitmap)} @@ -707,6 +800,10 @@ defmodule Module.Types.Descr do :empty -> acc end end + |> case do + [] -> 0 + acc -> acc + end end # Intersects two map literals; throws if their intersection is empty. @@ -776,6 +873,10 @@ defmodule Module.Types.Descr do end) end) end) + |> case do + [] -> 0 + acc -> acc + end end # Emptiness checking for maps. @@ -800,7 +901,7 @@ defmodule Module.Types.Descr do # The key is not shared between positive and negative maps, # and because the negative type is required, there is no value in common - tag == :closed and not optional?(neg_type) -> + tag == :closed and not is_optional_static(neg_type) -> false # The key is not shared between positive and negative maps, @@ -821,7 +922,7 @@ defmodule Module.Types.Descr do empty?(diff) or map_empty?(tag, Map.put(fields, key, diff), negs) %{} -> - if neg_tag == :closed and not optional?(type) do + if neg_tag == :closed and not is_optional_static(type) do false else # an absent key in a open negative map can be ignored @@ -835,19 +936,24 @@ defmodule Module.Types.Descr do # Takes a map dnf and a key and returns a list of unions of types # for that key. It has to traverse both fields and negative entries. defp map_split_on_key(dnf, key) do - Enum.flat_map(dnf, fn {tag, fields, negs} -> - # %{...} the open map in a positive intersection can be ignored - {fst, snd} = - if tag == :open and fields == %{} do - {term_or_optional(), term_or_optional()} - else - map_pop_key(tag, fields, key) + Enum.flat_map(dnf, fn + # Optimization: if there are no negatives, + # we can return the value directly. + {_tag, %{^key => value}, []} -> + [value] + + # Optimization: if there are no negatives + # and the key does not exist, return the default one. + {tag, %{}, []} -> + [map_tag_to_type(tag)] + + {tag, fields, negs} -> + {fst, snd} = map_pop_key(tag, fields, key) + + case map_split_negative(negs, key, []) do + :empty -> [] + negative -> negative |> pair_make_disjoint() |> pair_eliminate_negations(fst, snd) end - - case map_split_negative(negs, key, []) do - :empty -> [] - negative -> negative |> pair_make_disjoint() |> pair_eliminate_negations(fst, snd) - end end) end @@ -883,11 +989,11 @@ defmodule Module.Types.Descr do defp map_empty_negation?(tag, fields, {neg_tag, neg_fields}) do (tag == :closed and Enum.any?(neg_fields, fn {neg_key, neg_type} -> - not is_map_key(fields, neg_key) and not optional?(neg_type) + not is_map_key(fields, neg_key) and not is_optional_static(neg_type) end)) or (neg_tag == :closed and Enum.any?(fields, fn {key, type} -> - not is_map_key(neg_fields, key) and not optional?(type) + not is_map_key(neg_fields, key) and not is_optional_static(type) end)) end @@ -944,7 +1050,7 @@ defmodule Module.Types.Descr do keyword? = Inspect.List.keyword?(sorted) for {key, type} <- sorted, - not (tag == :open and optional?(type) and term_type?(type)) do + not (tag == :open and is_optional_static(type) and term_type?(type)) do key = if keyword? do {:__block__, [format: :keyword], [key]} @@ -953,7 +1059,7 @@ defmodule Module.Types.Descr do end cond do - not optional?(type) -> {key, to_quoted(type)} + not is_optional_static(type) -> {key, to_quoted(type)} empty?(type) -> {key, {:not_set, [], []}} true -> {key, {:if_set, [], [to_quoted(type)]}} end @@ -1096,7 +1202,7 @@ defmodule Module.Types.Descr do %{^key => v2} -> case fun.(key, v1, v2) do 0 -> acc - [] -> acc + value when value == @none -> acc value -> [{key, value} | acc] end diff --git a/lib/elixir/lib/module/types/expr.ex b/lib/elixir/lib/module/types/expr.ex index f0a2c1edea8..760650a9143 100644 --- a/lib/elixir/lib/module/types/expr.ex +++ b/lib/elixir/lib/module/types/expr.ex @@ -301,9 +301,15 @@ defmodule Module.Types.Expr do # TODO: fun.(args) def of_expr({{:., _meta1, [fun]}, _meta2, args}, stack, context) do - with {:ok, _fun_type, context} <- of_expr(fun, stack, context), - {:ok, _arg_types, context} <- + with {:ok, fun_type, context} <- of_expr(fun, stack, context), + {:ok, _args_types, context} <- map_reduce_ok(args, context, &of_expr(&1, stack, &2)) do + context = + case fun_fetch(fun_type, length(args)) do + :ok -> context + :error -> Of.incompatible_warn(fun, fun(), fun_type, stack, context) + end + {:ok, dynamic(), context} end end @@ -393,11 +399,7 @@ defmodule Module.Types.Expr do expected = if structs == [], do: @exception, else: Enum.reduce(structs, &union/2) formatter = fn expr -> - [ - "rescue #{expr_to_string(expr)} ->" |> indent(4), - ?\n, - format_hints(hints) - ] + {"rescue #{expr_to_string(expr)} ->", hints} end {:ok, _type, context} = Of.refine_var(var, expected, expr, formatter, stack, context) @@ -531,23 +533,28 @@ defmodule Module.Types.Expr do ## Warning formatting - def format_warning({:badupdate, type, expr, expected_type, actual_type, context}) do - [ - """ - incompatible types in #{type} update: + def format_diagnostic({:badupdate, type, expr, expected_type, actual_type, context}) do + traces = Of.collect_traces(expr, context) + + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + incompatible types in #{type} update: - #{expr_to_string(expr) |> indent(4)} + #{expr_to_string(expr) |> indent(4)} - expected type: + expected type: - #{to_quoted_string(expected_type) |> indent(4)} + #{to_quoted_string(expected_type) |> indent(4)} - but got type: + but got type: - #{to_quoted_string(actual_type) |> indent(4)} - """, - Of.format_traces(expr, context), - "\ntyping violation found at:" - ] + #{to_quoted_string(actual_type) |> indent(4)} + """, + Of.format_traces(traces) + ]) + } end end diff --git a/lib/elixir/lib/module/types/helpers.ex b/lib/elixir/lib/module/types/helpers.ex index a55e8fd65f2..cc62f46bff6 100644 --- a/lib/elixir/lib/module/types/helpers.ex +++ b/lib/elixir/lib/module/types/helpers.ex @@ -41,7 +41,9 @@ defmodule Module.Types.Helpers do """ #{hint()} when you rescue without specifying exception names, \ - the variable is assigned a type of a struct but all of its fields are unknown + the variable is assigned a type of a struct but all of its fields are unknown. \ + If you are trying to access an exception's :message key, either specify the \ + exception names or use `Exception.message/1`. """ end) end diff --git a/lib/elixir/lib/module/types/of.ex b/lib/elixir/lib/module/types/of.ex index 6b2b7a6f5f1..ab6c2a17a47 100644 --- a/lib/elixir/lib/module/types/of.ex +++ b/lib/elixir/lib/module/types/of.ex @@ -47,8 +47,7 @@ defmodule Module.Types.Of do # We need to return error otherwise it leads to cascading errors if empty?(new_type) do - {:error, - warn(__MODULE__, {:refine_var, old_type, type, var, context}, meta, stack, context)} + {:error, warn({:refine_var, old_type, type, var, context}, meta, stack, context)} else {:ok, new_type, context} end @@ -177,10 +176,15 @@ defmodule Module.Types.Of do # TODO: Use the struct default values to define the default types. def struct(struct, args_types, default_handling, meta, stack, context) do context = remote(struct, :__struct__, 0, meta, stack, context) + + info = + struct.__info__(:struct) || + raise "expected #{inspect(struct)} to return struct metadata, but got none" + term = term() defaults = - for %{field: field} <- struct.__info__(:struct), field != :__struct__ do + for %{field: field} <- info, field != :__struct__ do {field, term} end @@ -420,7 +424,7 @@ defmodule Module.Types.Of do meta = get_meta(expr) || stack.meta hints = if meta[:inferred_bitstring_spec], do: [:inferred_bitstring_spec], else: [] warning = {:incompatible, expr, expected_type, actual_type, hints, context} - warn(__MODULE__, warning, meta, stack, context) + warn(warning, meta, stack, context) end defp warn(warning, meta, stack, context) do @@ -429,15 +433,24 @@ defmodule Module.Types.Of do ## Traces - def format_traces(expr, %{vars: vars}) do + def collect_traces(expr, %{vars: vars}) do {_, versions} = Macro.prewalk(expr, %{}, fn {var_name, meta, var_context}, versions when is_atom(var_name) and is_atom(var_context) -> version = meta[:version] case vars do - %{^version => data} -> {:ok, Map.put(versions, version, data)} - %{} -> {:ok, versions} + %{^version => %{off_traces: [_ | _] = off_traces, name: name, context: context}} -> + {:ok, + Map.put(versions, version, %{ + type: :variable, + name: name, + context: context, + traces: collect_var_traces(off_traces) + })} + + _ -> + {:ok, versions} end node, versions -> @@ -447,40 +460,55 @@ defmodule Module.Types.Of do versions |> Map.values() |> Enum.sort_by(& &1.name) - |> Enum.map(&format_trace/1) end - defp format_trace(%{off_traces: []}) do - [] + defp collect_var_traces(traces) do + traces + |> Enum.reverse() + |> Enum.map(fn {expr, file, type, formatter} -> + meta = get_meta(expr) + + {formatted_expr, formatter_hints} = + case formatter do + :default -> {expr_to_string(expr), []} + formatter -> formatter.(expr) + end + + %{ + file: file, + meta: meta, + formatted_expr: formatted_expr, + formatted_hints: format_hints(formatter_hints ++ expr_hints(expr)), + formatted_type: to_quoted_string(type) + } + end) end - defp format_trace(%{name: name, context: context, off_traces: traces}) do - traces = - traces - |> Enum.reverse() - |> Enum.map(fn {expr, file, type, formatter} -> - meta = get_meta(expr) + def format_traces(traces) do + Enum.map(traces, &format_trace/1) + end + defp format_trace(%{type: :variable, name: name, context: context, traces: traces}) do + traces = + for trace <- traces do location = - file + trace.file |> Path.relative_to_cwd() - |> Exception.format_file_line(meta[:line]) + |> Exception.format_file_line(trace.meta[:line]) |> String.replace_suffix(":", "") [ """ - # type: #{to_quoted_string(type) |> indent(4)} + # type: #{indent(trace.formatted_type, 4)} # from: #{location} \ """, - case formatter do - :default -> [expr |> expr_to_string() |> indent(4), ?\n] - formatter -> formatter.(expr) - end, - format_hints(expr_hints(expr)) + indent(trace.formatted_expr, 4), + ?\n, + trace.formatted_hints ] - end) + end type_or_types = pluralize(traces, "type", "types") ["\nwhere #{format_var(name, context)} was given the #{type_or_types}:\n" | traces] @@ -500,96 +528,169 @@ defmodule Module.Types.Of do ## Warning formatting - def format_warning({:refine_var, old_type, new_type, var, context}) do - [ - """ - incompatible types assigned to #{format_var(var)}: + def format_diagnostic({:refine_var, old_type, new_type, var, context}) do + traces = collect_traces(var, context) + + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + incompatible types assigned to #{format_var(var)}: + + #{to_quoted_string(old_type)} !~ #{to_quoted_string(new_type)} + """, + format_traces(traces) + ]) + } + end + + def format_diagnostic({:incompatible, expr, expected_type, actual_type, hints, context}) do + traces = collect_traces(expr, context) + + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + incompatible types in expression: + + #{expr_to_string(expr) |> indent(4)} + + expected type: + + #{to_quoted_string(expected_type) |> indent(4)} - #{to_quoted_string(old_type)} !~ #{to_quoted_string(new_type)} - """, - format_traces(var, context), - "\ntyping violation found at:" - ] + but got type: + + #{to_quoted_string(actual_type) |> indent(4)} + """, + format_traces(traces), + format_hints(hints) + ]) + } end - def format_warning({:incompatible, expr, expected_type, actual_type, hints, context}) do - [ - """ - incompatible types in expression: + def format_diagnostic({:badmap, expr, type, key, context}) do + traces = collect_traces(expr, context) + + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + expected a map or struct when accessing .#{key} in expression: - #{expr_to_string(expr) |> indent(4)} + #{expr_to_string(expr) |> indent(4)} + """, + empty_if(dot_var?(expr), """ - expected type: + but got type: - #{to_quoted_string(expected_type) |> indent(4)} + #{to_quoted_string(type) |> indent(4)} + """), + format_traces(traces), + format_hints([:dot]) + ]) + } + end + + def format_diagnostic({:badkey, expr, type, key, context}) do + traces = collect_traces(expr, context) + + %{ + details: %{typing_traces: traces}, + span: expr |> get_meta() |> :elixir_env.calculate_span(key) |> Keyword.get(:span), + message: + IO.iodata_to_binary([ + """ + unknown key .#{key} in expression: + + #{expr_to_string(expr) |> indent(4)} + """, + empty_if(dot_var?(expr), """ - but got type: + the given type does not have the given key: - #{to_quoted_string(actual_type) |> indent(4)} - """, - format_traces(expr, context), - format_hints(hints), - "\ntyping violation found at:" - ] + #{to_quoted_string(type) |> indent(4)} + """), + format_traces(traces) + ]) + } end - def format_warning({:badmap, expr, type, key, context}) do - [ - """ - expected a map or struct when accessing .#{key} in expression: + def format_diagnostic({:badmodule, expr, type, fun, arity, hints, context}) do + traces = collect_traces(expr, context) - #{expr_to_string(expr) |> indent(4)} - """, - empty_if(dot_var?(expr), """ + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + expected a module (an atom) when invoking #{fun}/#{arity} in expression: + + #{expr_to_string(expr) |> indent(4)} + """, + empty_if(dot_var?(expr), """ - but got type: + but got type: - #{to_quoted_string(type) |> indent(4)} - """), - format_traces(expr, context), - format_hints([:dot]), - "\ntyping violation found at:" - ] + #{to_quoted_string(type) |> indent(4)} + """), + format_traces(traces), + format_hints(hints) + ]) + } end - def format_warning({:badkey, expr, type, key, context}) do - [ - """ - unknown key .#{key} in expression: + def format_diagnostic({:mismatched_comparison, expr, context}) do + traces = collect_traces(expr, context) - #{expr_to_string(expr) |> indent(4)} - """, - empty_if(dot_var?(expr), """ + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + comparison between incompatible types found: - the given type does not have the given key: + #{expr_to_string(expr) |> indent(4)} + """, + format_traces(traces), + """ - #{to_quoted_string(type) |> indent(4)} - """), - format_traces(expr, context), - "\ntyping violation found at:" - ] + While Elixir can compare across all types, you are comparing \ + across types which are always distinct, and the result is either \ + always true or always false + """ + ]) + } end - def format_warning({:badmodule, expr, type, fun, arity, hints, context}) do - [ - """ - expected a module (an atom) when invoking #{fun}/#{arity} in expression: + def format_diagnostic({:struct_comparison, expr, context}) do + traces = collect_traces(expr, context) - #{expr_to_string(expr) |> indent(4)} - """, - empty_if(dot_var?(expr), """ + %{ + details: %{typing_traces: traces}, + message: + IO.iodata_to_binary([ + """ + comparison with structs found: - but got type: + #{expr_to_string(expr) |> indent(4)} + """, + format_traces(traces), + """ - #{to_quoted_string(type) |> indent(4)} - """), - format_traces(expr, context), - format_hints(hints), - "\ntyping violation found at:" - ] + Comparison operators (>, <, >=, <=, min, and max) perform structural \ + and not semantic comparison. Comparing with a struct won't give meaningful \ + results. Structs that can be compared typically define a compare/2 function \ + within their modules that can be used for semantic comparison. + """ + ]) + } end - def format_warning({:undefined_module, module, fun, arity}) do + def format_diagnostic({:undefined_module, module, fun, arity}) do top = if fun == :__struct__ and arity == 0 do "struct #{inspect(module)}" @@ -597,78 +698,61 @@ defmodule Module.Types.Of do Exception.format_mfa(module, fun, arity) end - [ - top, - " is undefined (module ", - inspect(module), - " is not available or is yet to be defined)" - ] - end - - def format_warning({:undefined_function, module, :__struct__, 0, _exports}) do - "struct #{inspect(module)} is undefined (there is such module but it does not define a struct)" - end - - def format_warning({:undefined_function, module, fun, arity, exports}) do - [ - Exception.format_mfa(module, fun, arity), - " is undefined or private", - UndefinedFunctionError.hint_for_loaded_module(module, fun, arity, exports) - ] - end - - def format_warning({:deprecated, module, fun, arity, reason}) do - [ - Exception.format_mfa(module, fun, arity), - " is deprecated. ", - reason - ] - end - - def format_warning({:unrequired_module, module, fun, arity}) do - [ - "you must require ", - inspect(module), - " before invoking the macro ", - Exception.format_mfa(module, fun, arity) - ] - end - - def format_warning({:mismatched_comparison, expr, context}) do - [ - """ - comparison between incompatible types found: - - #{expr_to_string(expr) |> indent(4)} - """, - format_traces(expr, context), - """ - - While Elixir can compare across all types, you are comparing \ - across types which are always distinct, and the result is either \ - always true or always false - """, - "\ntyping violation found at:" - ] - end - - def format_warning({:struct_comparison, expr, context}) do - [ - """ - comparison with structs found: - - #{expr_to_string(expr) |> indent(4)} - """, - format_traces(expr, context), - """ - - Comparison operators (>, <, >=, <=, min, and max) perform structural \ - and not semantic comparison. Comparing with a struct won't give meaningful \ - results. Struct that can be compared typically define a compare/2 function \ - within their modules that can be used for semantic comparison - """, - "\ntyping violation found at:" - ] + %{ + message: + IO.iodata_to_binary([ + top, + " is undefined (module ", + inspect(module), + " is not available or is yet to be defined)" + ]), + group: true + } + end + + def format_diagnostic({:undefined_function, module, :__struct__, 0, _exports}) do + %{ + message: + "struct #{inspect(module)} is undefined (there is such module but it does not define a struct)", + group: true + } + end + + def format_diagnostic({:undefined_function, module, fun, arity, exports}) do + %{ + message: + IO.iodata_to_binary([ + Exception.format_mfa(module, fun, arity), + " is undefined or private", + UndefinedFunctionError.hint_for_loaded_module(module, fun, arity, exports) + ]), + group: true + } + end + + def format_diagnostic({:deprecated, module, fun, arity, reason}) do + %{ + message: + IO.iodata_to_binary([ + Exception.format_mfa(module, fun, arity), + " is deprecated. ", + reason + ]), + group: true + } + end + + def format_diagnostic({:unrequired_module, module, fun, arity}) do + %{ + message: + IO.iodata_to_binary([ + "you must require ", + inspect(module), + " before invoking the macro ", + Exception.format_mfa(module, fun, arity) + ]), + group: true + } end defp dot_var?(expr) do diff --git a/lib/elixir/lib/regex.ex b/lib/elixir/lib/regex.ex index 83d88479336..d53b2010453 100644 --- a/lib/elixir/lib/regex.ex +++ b/lib/elixir/lib/regex.ex @@ -186,27 +186,7 @@ defmodule Regex do defstruct re_pattern: nil, source: "", opts: [], re_version: "" - @type re_option :: - :unicode - | :caseless - | :dotall - | :multiline - | :extended - | :firstline - | :ungreedy - | :anchored - | :dollar_endonly - | :no_auto_capture - | :newline - - @type t :: %__MODULE__{re_pattern: term, source: binary, opts: binary | [re_option()]} - - @type capture_option :: - :all | :first | :all_but_first | :none | :all_names | [binary() | atom()] - @type run_option :: - {:return, :binary | :index} - | {:capture, capture_option()} - | {:offset, non_neg_integer()} + @type t :: %__MODULE__{re_pattern: term, source: binary, opts: binary | [term]} defmodule CompileError do @moduledoc """ @@ -242,7 +222,7 @@ defmodule Regex do {:ok, Regex.compile!("foo", [:caseless])} """ - @spec compile(binary, binary | [re_option()]) :: {:ok, t} | {:error, any} + @spec compile(binary, binary | [term]) :: {:ok, t} | {:error, any} def compile(source, opts \\ "") when is_binary(source) do compile(source, opts, version()) end @@ -270,7 +250,7 @@ defmodule Regex do @doc """ Compiles the regular expression and raises `Regex.CompileError` in case of errors. """ - @spec compile!(binary, binary | [re_option()]) :: t + @spec compile!(binary, binary | [term]) :: t def compile!(source, options \\ "") when is_binary(source) do case compile(source, options) do {:ok, regex} -> regex @@ -377,7 +357,7 @@ defmodule Regex do ["d", ""] """ - @spec run(t, binary, [run_option()]) :: nil | [binary] | [{integer, integer}] + @spec run(t, binary, [term]) :: nil | [binary] | [{integer, integer}] def run(regex, string, options \\ []) def run(%Regex{} = regex, string, options) when is_binary(string) do @@ -412,7 +392,7 @@ defmodule Regex do nil """ - @spec named_captures(t, String.t(), [{:return, :binary | :index}]) :: map | nil + @spec named_captures(t, String.t(), [term]) :: map | nil def named_captures(regex, string, options \\ []) when is_binary(string) do names = names(regex) options = Keyword.put(options, :capture, names) @@ -456,7 +436,7 @@ defmodule Regex do [:caseless] """ - @spec opts(t) :: [re_option()] + @spec opts(t) :: [term] def opts(%Regex{opts: opts}) do opts end @@ -528,7 +508,7 @@ defmodule Regex do [["cd"], ["ce"]] """ - @spec scan(t(), String.t(), [run_option()]) :: [[String.t()]] | [[{integer(), integer()}]] + @spec scan(t(), String.t(), [term()]) :: [[String.t()]] | [[{integer(), integer()}]] def scan(regex, string, options \\ []) def scan(%Regex{} = regex, string, options) when is_binary(string) do @@ -615,12 +595,7 @@ defmodule Regex do ["a", "b", "c"] """ - @spec split(t, String.t(), - parts: pos_integer() | :infinity, - trim: boolean(), - on: capture_option(), - include_captures: boolean() - ) :: [String.t()] + @spec split(t, String.t(), [term]) :: [String.t()] def split(regex, string, options \\ []) def split(%Regex{}, "", opts) do diff --git a/lib/elixir/lib/registry.ex b/lib/elixir/lib/registry.ex index 418bc234622..c3fa949f75c 100644 --- a/lib/elixir/lib/registry.ex +++ b/lib/elixir/lib/registry.ex @@ -757,28 +757,28 @@ defmodule Registry do In the example below we register the current process and look it up both from itself and other processes: - iex> Registry.start_link(keys: :unique, name: Registry.UniqueLookupTest) - iex> Registry.values(Registry.UniqueLookupTest, "hello", self()) + iex> Registry.start_link(keys: :unique, name: Registry.UniqueValuesTest) + iex> Registry.values(Registry.UniqueValuesTest, "hello", self()) [] - iex> {:ok, _} = Registry.register(Registry.UniqueLookupTest, "hello", :world) - iex> Registry.values(Registry.UniqueLookupTest, "hello", self()) + iex> {:ok, _} = Registry.register(Registry.UniqueValuesTest, "hello", :world) + iex> Registry.values(Registry.UniqueValuesTest, "hello", self()) [:world] - iex> Task.async(fn -> Registry.values(Registry.UniqueLookupTest, "hello", self()) end) |> Task.await() + iex> Task.async(fn -> Registry.values(Registry.UniqueValuesTest, "hello", self()) end) |> Task.await() [] iex> parent = self() - iex> Task.async(fn -> Registry.values(Registry.UniqueLookupTest, "hello", parent) end) |> Task.await() + iex> Task.async(fn -> Registry.values(Registry.UniqueValuesTest, "hello", parent) end) |> Task.await() [:world] The same applies to duplicate registries: - iex> Registry.start_link(keys: :duplicate, name: Registry.DuplicateLookupTest) - iex> Registry.values(Registry.DuplicateLookupTest, "hello", self()) + iex> Registry.start_link(keys: :duplicate, name: Registry.DuplicateValuesTest) + iex> Registry.values(Registry.DuplicateValuesTest, "hello", self()) [] - iex> {:ok, _} = Registry.register(Registry.DuplicateLookupTest, "hello", :world) - iex> Registry.values(Registry.DuplicateLookupTest, "hello", self()) + iex> {:ok, _} = Registry.register(Registry.DuplicateValuesTest, "hello", :world) + iex> Registry.values(Registry.DuplicateValuesTest, "hello", self()) [:world] - iex> {:ok, _} = Registry.register(Registry.DuplicateLookupTest, "hello", :another) - iex> Enum.sort(Registry.values(Registry.DuplicateLookupTest, "hello", self())) + iex> {:ok, _} = Registry.register(Registry.DuplicateValuesTest, "hello", :another) + iex> Enum.sort(Registry.values(Registry.DuplicateValuesTest, "hello", self())) [:another, :world] """ diff --git a/lib/elixir/lib/supervisor.ex b/lib/elixir/lib/supervisor.ex index 0307673567d..2507ef98e79 100644 --- a/lib/elixir/lib/supervisor.ex +++ b/lib/elixir/lib/supervisor.ex @@ -595,7 +595,7 @@ defmodule Supervisor do @typedoc "The supervisor reference." @type supervisor :: pid | name | {atom, node} - @typedoc "Options given to `start_link/2` and `c:init/1`." + @typedoc "Options given to `start_link/2` and `init/2`." @type init_option :: {:strategy, strategy} | {:max_restarts, non_neg_integer} diff --git a/lib/elixir/lib/system.ex b/lib/elixir/lib/system.ex index 72579414ea7..24e3c457b6b 100644 --- a/lib/elixir/lib/system.ex +++ b/lib/elixir/lib/system.ex @@ -1054,7 +1054,7 @@ defmodule System do * `:arg0` - sets the command arg0 * `:stderr_to_stdout` - redirects stderr to stdout when `true`, no effect - if `use_stdio` is `false``. + if `use_stdio` is `false`. * `:use_stdio` - `true` by default, setting it to false allows direct interaction with the terminal from the callee @@ -1239,7 +1239,7 @@ defmodule System do This time is monotonically increasing and starts in an unspecified point in time. """ - @spec monotonic_time(time_unit) :: integer + @spec monotonic_time(time_unit | :native) :: integer def monotonic_time(unit) do :erlang.monotonic_time(normalize_time_unit(unit)) end @@ -1265,7 +1265,7 @@ defmodule System do case of time warps although the VM works towards aligning them. This time is not monotonic. """ - @spec system_time(time_unit) :: integer + @spec system_time(time_unit | :native) :: integer def system_time(unit) do :erlang.system_time(normalize_time_unit(unit)) end @@ -1316,7 +1316,7 @@ defmodule System do `monotonic_time/1`), gives the Erlang system time that corresponds to that monotonic time. """ - @spec time_offset(time_unit) :: integer + @spec time_offset(time_unit | :native) :: integer def time_offset(unit) do :erlang.time_offset(normalize_time_unit(unit)) end diff --git a/lib/elixir/pages/anti-patterns/design-anti-patterns.md b/lib/elixir/pages/anti-patterns/design-anti-patterns.md index 5423682cfc8..68fb3f45428 100644 --- a/lib/elixir/pages/anti-patterns/design-anti-patterns.md +++ b/lib/elixir/pages/anti-patterns/design-anti-patterns.md @@ -17,12 +17,12 @@ defmodule AlternativeInteger do @spec parse(String.t(), keyword()) :: integer() | {integer(), String.t()} | :error def parse(string, options \\ []) when is_list(options) do if Keyword.get(options, :discard_rest, false) do - Integer.parse(string) - else case Integer.parse(string) do {int, _rest} -> int :error -> :error end + else + Integer.parse(string) end end end @@ -30,11 +30,11 @@ end ```elixir iex> AlternativeInteger.parse("13") -13 -iex> AlternativeInteger.parse("13", discard_rest: true) -13 +{13, ""} iex> AlternativeInteger.parse("13", discard_rest: false) {13, ""} +iex> AlternativeInteger.parse("13", discard_rest: true) +13 ``` #### Refactoring diff --git a/lib/elixir/pages/getting-started/binaries-strings-and-charlists.md b/lib/elixir/pages/getting-started/binaries-strings-and-charlists.md index 9fcd587e179..3e0a22a4f5f 100644 --- a/lib/elixir/pages/getting-started/binaries-strings-and-charlists.md +++ b/lib/elixir/pages/getting-started/binaries-strings-and-charlists.md @@ -9,7 +9,9 @@ iex> is_binary(string) true ``` -In this chapter, we will gain clarity on what exactly binaries are, how they relate to strings, and what single-quoted values, `'like this'`, mean in Elixir. Although strings are one of the most common data types in computer languages, they are subtly complex and are often misunderstood. To understand strings in Elixir, we have to educate ourselves about [Unicode](https://en.wikipedia.org/wiki/Unicode) and character encodings, specifically the [UTF-8](https://en.wikipedia.org/wiki/UTF-8) encoding. +In this chapter, we will gain clarity on what exactly binaries are and how they relate to strings. We will also learn about charlists, `~c"like this"`, which are often used for interoperability with Erlang. + +Although strings are one of the most common data types in computer languages, they are subtly complex and are often misunderstood. To understand strings in Elixir, let's first discuss [Unicode](https://en.wikipedia.org/wiki/Unicode) and character encodings, specifically the [UTF-8](https://en.wikipedia.org/wiki/UTF-8) encoding. ## Unicode and Code Points diff --git a/lib/elixir/pages/mix-and-otp/genservers.md b/lib/elixir/pages/mix-and-otp/genservers.md index 734c810b7c2..0c726da3788 100644 --- a/lib/elixir/pages/mix-and-otp/genservers.md +++ b/lib/elixir/pages/mix-and-otp/genservers.md @@ -124,7 +124,7 @@ iex> {:ok, registry} = GenServer.start_link(KV.Registry, :ok) {:ok, #PID<0.136.0>} iex> GenServer.cast(registry, {:create, "shopping"}) :ok -iex> {:ok, bk} = GenServer.call(registry, {:lookup, "shopping"}) +iex> {:ok, bucket} = GenServer.call(registry, {:lookup, "shopping"}) {:ok, #PID<0.174.0>} ``` diff --git a/lib/elixir/pages/references/compatibility-and-deprecations.md b/lib/elixir/pages/references/compatibility-and-deprecations.md index 8611fe28f2e..5b7b7f40058 100644 --- a/lib/elixir/pages/references/compatibility-and-deprecations.md +++ b/lib/elixir/pages/references/compatibility-and-deprecations.md @@ -8,12 +8,11 @@ Elixir applies bug fixes only to the latest minor branch. Security patches are a Elixir version | Support :------------- | :----------------------------- -1.17 | Development -1.16 | Bug fixes and security patches +1.17 | Bug fixes and security patches +1.16 | Security patches only 1.15 | Security patches only 1.14 | Security patches only 1.13 | Security patches only -1.12 | Security patches only New releases are announced in the read-only [announcements mailing list](https://groups.google.com/group/elixir-lang-ann). All security releases [will be tagged with `[security]`](https://groups.google.com/forum/#!searchin/elixir-lang-ann/%5Bsecurity%5D%7Csort:date). @@ -214,4 +213,4 @@ Version | Deprecated feature | Replaced by (ava [v1.14]: https://github.com/elixir-lang/elixir/blob/v1.14/CHANGELOG.md#4-hard-deprecations [v1.15]: https://github.com/elixir-lang/elixir/blob/v1.15/CHANGELOG.md#4-hard-deprecations [v1.16]: https://github.com/elixir-lang/elixir/blob/v1.16/CHANGELOG.md#4-hard-deprecations -[v1.17]: https://github.com/elixir-lang/elixir/blob/main/CHANGELOG.md#4-hard-deprecations +[v1.17]: https://github.com/elixir-lang/elixir/blob/v1.17/CHANGELOG.md#4-hard-deprecations diff --git a/lib/elixir/pages/references/gradual-set-theoretic-types.md b/lib/elixir/pages/references/gradual-set-theoretic-types.md index daaaed6ef95..a2ace46cf6d 100644 --- a/lib/elixir/pages/references/gradual-set-theoretic-types.md +++ b/lib/elixir/pages/references/gradual-set-theoretic-types.md @@ -72,7 +72,7 @@ The current milestone is to implement type inference and type checking of Elixir If the results are satisfactory, the next milestone will include a mechanism for defining typed structs. Elixir programs frequently pattern match on structs, which reveals information about the struct fields, but it knows nothing about their respective types. By propagating types from structs and their fields throughout the program, we will increase the type system’s ability to find errors while further straining our type system implementation. Proposals including the required changes to the language surface will be sent to the community once we reach this stage. -The third milestone is to introduce set-theoretic type signatures for functions. Unfortunately, the existing typespecs are not precise enough for set-theoretic types and they will be phased out of the language and moved into a separate library once this stage concludes. +The third milestone is to introduce set-theoretic type signatures for functions. Unfortunately, the existing Erlang Typespecs are not precise enough for set-theoretic types and they will be phased out of the language and have their postprocessing moved into a separate library once this stage concludes. ## Acknowledgements diff --git a/lib/elixir/scripts/docs_config.exs b/lib/elixir/scripts/docs_config.exs index b161917480e..43155b6801e 100644 --- a/lib/elixir/scripts/docs_config.exs +++ b/lib/elixir/scripts/docs_config.exs @@ -5,10 +5,13 @@ skipped = Version.parse!("1.0.3") list_contents = - text_tags - |> String.split() - |> Enum.map(fn "v" <> rest -> Version.parse!(rest) end) - |> Enum.filter(&(Version.compare(&1, skipped) == :gt)) + for( + "v" <> rest <- String.split(text_tags), + not String.ends_with?(rest, "-latest"), + version = Version.parse!(rest), + Version.compare(version, skipped) == :gt, + do: version + ) |> Enum.sort({:desc, Version}) |> Enum.map_intersperse(", ", fn version -> version_string = Version.to_string(version) diff --git a/lib/elixir/src/elixir_dispatch.erl b/lib/elixir/src/elixir_dispatch.erl index bcf8e841b6e..12e753709c8 100644 --- a/lib/elixir/src/elixir_dispatch.erl +++ b/lib/elixir/src/elixir_dispatch.erl @@ -77,7 +77,7 @@ import_function(Meta, Name, Arity, E) -> false; {import, Receiver} -> require_function(Meta, Receiver, Name, Arity, E); - {ambiguous, Ambiguous} -> + {ambiguous, _} = Ambiguous -> elixir_errors:file_error(Meta, E, ?MODULE, {import, Ambiguous, Name, Arity}); false -> case elixir_import:special_form(Name, Arity) of @@ -131,7 +131,8 @@ dispatch_import(Meta, Name, Args, S, E, Callback) -> case expand_import(Meta, Name, Arity, E, [], AllowLocals, true) of {macro, Receiver, Expander} -> check_deprecated(macro, Meta, Receiver, Name, Arity, E), - expand_quoted(Meta, Receiver, Name, Arity, Expander(Args, S), S, E); + Caller = {?line(Meta), S, E}, + expand_quoted(Meta, Receiver, Name, Arity, Expander(Args, Caller), S, E); {function, Receiver, NewName} -> elixir_expand:expand({{'.', Meta, [Receiver, NewName]}, Meta, Args}, S, E); not_found -> @@ -151,7 +152,8 @@ dispatch_require(Meta, Receiver, Name, Args, S, E, Callback) when is_atom(Receiv case expand_require(Meta, Receiver, Name, Arity, E, true) of {macro, Receiver, Expander} -> check_deprecated(macro, Meta, Receiver, Name, Arity, E), - expand_quoted(Meta, Receiver, Name, Arity, Expander(Args, S), S, E); + Caller = {?line(Meta), S, E}, + expand_quoted(Meta, Receiver, Name, Arity, Expander(Args, Caller), S, E); error -> check_deprecated(function, Meta, Receiver, Name, Arity, E), elixir_env:trace({remote_function, Meta, Receiver, Name, Arity}, E), @@ -246,26 +248,23 @@ expand_require(Required, Meta, Receiver, Name, Arity, E, Trace) -> %% Expansion helpers expander_macro_fun(Meta, Fun, Receiver, Name, E) -> - fun(Args, S) -> expand_macro_fun(Meta, Fun, Receiver, Name, Args, S, E) end. + fun(Args, Caller) -> expand_macro_fun(Meta, Fun, Receiver, Name, Args, Caller, E) end. expander_macro_named(Meta, Receiver, Name, Arity, E) -> ProperName = elixir_utils:macro_name(Name), ProperArity = Arity + 1, Fun = fun Receiver:ProperName/ProperArity, - fun(Args, S) -> expand_macro_fun(Meta, Fun, Receiver, Name, Args, S, E) end. - -expand_macro_fun(Meta, Fun, Receiver, Name, Args, S, E) -> - Line = ?line(Meta), - EArg = {Line, S, E}, + fun(Args, Caller) -> expand_macro_fun(Meta, Fun, Receiver, Name, Args, Caller, E) end. +expand_macro_fun(Meta, Fun, Receiver, Name, Args, Caller, E) -> try - apply(Fun, [EArg | Args]) + apply(Fun, [Caller | Args]) catch Kind:Reason:Stacktrace -> Arity = length(Args), MFA = {Receiver, elixir_utils:macro_name(Name), Arity+1}, - Info = [{Receiver, Name, Arity, [{file, "expanding macro"}]}, caller(Line, E)], - erlang:raise(Kind, Reason, prune_stacktrace(Stacktrace, MFA, Info, {ok, EArg})) + Info = [{Receiver, Name, Arity, [{file, "expanding macro"}]}, caller(?line(Meta), E)], + erlang:raise(Kind, Reason, prune_stacktrace(Stacktrace, MFA, Info, {ok, Caller})) end. expand_quoted(Meta, Receiver, Name, Arity, Quoted, S, E) -> @@ -342,7 +341,7 @@ is_import(Meta, Arity) -> end. % %% We've reached the macro wrapper fun, skip it with the rest -prune_stacktrace([{_, _, [E | _], _} | _], _MFA, Info, {ok, E}) -> +prune_stacktrace([{_, _, [Caller | _], _} | _], _MFA, Info, {ok, Caller}) -> Info; %% We've reached the invoked macro, skip it prune_stacktrace([{M, F, A, _} | _], {M, F, A}, Info, _E) -> diff --git a/lib/elixir/src/elixir_env.erl b/lib/elixir/src/elixir_env.erl index 93dc5e90293..372303199fc 100644 --- a/lib/elixir/src/elixir_env.erl +++ b/lib/elixir/src/elixir_env.erl @@ -32,7 +32,7 @@ trace(Event, #{tracers := Tracers} = E) -> to_caller({Line, #elixir_ex{vars={Read, _}}, Env}) -> Env#{line := Line, versioned_vars := Read}; -to_caller(#{} = Env) -> +to_caller(#{'__struct__' := 'Elixir.Macro.Env'} = Env) -> Env. with_vars(Env, Vars) when is_list(Vars) -> @@ -91,7 +91,7 @@ reset_unused_vars(#elixir_ex{unused={_Unused, Version}} = S) -> check_unused_vars(#elixir_ex{unused={Unused, _Version}}, E) -> [elixir_errors:file_warn(calculate_span(Meta, Name), E, ?MODULE, {unused_var, Name, Overridden}) || - {{{Name, nil}, _}, {Meta, Overridden}} <- maps:to_list(Unused), is_unused_var(Name)], + {{{Name, _Kind}, _Count}, {Meta, Overridden}} <- maps:to_list(Unused), is_unused_var(Name)], E. calculate_span(Meta, Name) -> @@ -120,8 +120,8 @@ merge_and_check_unused_vars(Current, Unused, ClauseUnused, E) -> Acc end; - ({{Name, Kind}, _Count}, {Meta, Overridden}, Acc) -> - case (Kind == nil) andalso is_unused_var(Name) of + ({{Name, _Kind}, _Count}, {Meta, Overridden}, Acc) -> + case is_unused_var(Name) of true -> Warn = {unused_var, Name, Overridden}, elixir_errors:file_warn(Meta, E, ?MODULE, Warn); diff --git a/lib/elixir/src/elixir_erl_pass.erl b/lib/elixir/src/elixir_erl_pass.erl index 89c0409e260..c44bf3c62df 100644 --- a/lib/elixir/src/elixir_erl_pass.erl +++ b/lib/elixir/src/elixir_erl_pass.erl @@ -425,16 +425,16 @@ translate_with_else(Meta, [{'else', Else}], S) -> RaiseClause = {'->', Generated, [[RaiseVar], RaiseExpr]}, Clauses = elixir_erl_clauses:get_clauses('else', [{'else', Else ++ [RaiseClause]}], match), - {TranslatedClauses, SC} = elixir_erl_clauses:clauses(Clauses, SV), - with_else_closure(Meta, TranslatedClauses, SC). + {TranslatedClauses, SC} = elixir_erl_clauses:clauses(Clauses, SV#elixir_erl{extra=pin_guard}), + with_else_closure(Generated, TranslatedClauses, SC#elixir_erl{extra=SV#elixir_erl.extra}). with_else_closure(Meta, TranslatedClauses, S) -> Ann = ?ann(Meta), {_, FunErlVar, SC} = elixir_erl_var:assign(Meta, S), {_, ArgErlVar, SA} = elixir_erl_var:assign(Meta, SC), - FunAssign = {match, Ann, FunErlVar, {'fun', Ann, {clauses, TranslatedClauses}}}, - FunCall = {call, Ann, FunErlVar, [ArgErlVar]}, Generated = erl_anno:set_generated(true, Ann), + FunAssign = {match, Ann, FunErlVar, {'fun', Generated, {clauses, TranslatedClauses}}}, + FunCall = {call, Ann, FunErlVar, [ArgErlVar]}, {{clause, Generated, [ArgErlVar], [], [FunCall]}, FunAssign, SA}. translate_with_do([{'<-', Meta, [{Var, _, Ctx} = Left, Expr]} | Rest], Ann, Do, Else, S) when is_atom(Var), is_atom(Ctx) -> diff --git a/lib/elixir/src/elixir_errors.erl b/lib/elixir/src/elixir_errors.erl index e925be8491b..e73439c6bae 100644 --- a/lib/elixir/src/elixir_errors.erl +++ b/lib/elixir/src/elixir_errors.erl @@ -6,12 +6,11 @@ -module(elixir_errors). -export([compile_error/1, compile_error/3, parse_error/5]). -export([function_error/4, module_error/4, file_error/4]). --export([format_snippet/7]). +-export([format_snippet/6]). -export([erl_warn/3, file_warn/4]). -export([prefix/1]). --export([print_diagnostic/2, emit_diagnostic/6]). --export([print_warning/2, print_warning/3]). --export([print_warning_group/2]). +-export([print_diagnostics/1, print_diagnostic/2, emit_diagnostic/6]). +-export([print_warning/3]). -include("elixir.hrl"). -type location() :: non_neg_integer() | {non_neg_integer(), non_neg_integer()}. @@ -20,35 +19,9 @@ %% TODO: Remove me on Elixir v2.0. %% Called by deprecated Kernel.ParallelCompiler.print_warning. print_warning(Position, File, Message) -> - Output = format_snippet(Position, File, Message, nil, warning, [], nil), + Output = format_snippet(warning, Position, File, Message, nil, #{}), io:put_chars(standard_error, [Output, $\n, $\n]). -%% Called by Module.ParallelChecker. -print_warning(Message, Diagnostic) -> - #{file := File, position := Position, stacktrace := S} = Diagnostic, - Snippet = read_snippet(File, Position), - Span = get_span(Diagnostic), - Output = format_snippet(Position, File, Message, Snippet, warning, S, Span), - io:put_chars(standard_error, [Output, $\n, $\n]). - -%% Called by Module.ParallelChecker. -print_warning_group(Message, [Diagnostic | Others]) -> - #{file := File, position := Position, stacktrace := S} = Diagnostic, - Snippet = read_snippet(File, Position), - Span = get_span(Diagnostic), - Formatted = format_snippet(Position, File, Message, Snippet, warning, S, Span), - LineNumber = extract_line(Position), - LineDigits = get_line_number_digits(LineNumber, 1), - Padding = case Snippet of - nil -> 0; - _ -> max(4, LineDigits + 2) - end, - Locations = [["\n", n_spaces(Padding), "└─ ", 'Elixir.Exception':format_stacktrace_entry(ES)] || #{stacktrace := [ES]} <- Others], - io:put_chars(standard_error, [Formatted, Locations, $\n, $\n]). - -get_span(#{span := nil}) -> nil; -get_span(#{span := Span}) -> Span. - read_snippet(nil, _Position) -> nil; read_snippet(<<"nofile">>, _Position) -> @@ -77,20 +50,33 @@ traverse_file_line(IoDevice, N) -> file:read_line(IoDevice), traverse_file_line(IoDevice, N - 1). -print_diagnostic(#{severity := Severity, message := M, stacktrace := Stacktrace, position := P, file := F} = Diagnostic, ReadSnippet) -> +%% Used by Module.ParallelChecker. +print_diagnostics([Diagnostic | Others]) -> + #{file := File, position := Position, message := Message} = Diagnostic, + Snippet = read_snippet(File, Position), + Formatted = format_snippet(warning, Position, File, Message, Snippet, Diagnostic), + LineNumber = extract_line(Position), + LineDigits = get_line_number_digits(LineNumber, 1), + Padding = case Snippet of + nil -> 0; + _ -> max(4, LineDigits + 2) + end, + Locations = [["\n", n_spaces(Padding), "└─ ", 'Elixir.Exception':format_stacktrace_entry(ES)] || #{stacktrace := [ES]} <- Others], + io:put_chars(standard_error, [Formatted, Locations, $\n, $\n]). + +print_diagnostic(#{severity := S, message := M, position := P, file := F} = Diagnostic, ReadSnippet) -> Snippet = case ReadSnippet of true -> read_snippet(F, P); false -> nil end, - Span = get_span(Diagnostic), - Output = format_snippet(P, F, M, Snippet, Severity, Stacktrace, Span), + Output = format_snippet(S, P, F, M, Snippet, Diagnostic), MaybeStack = case (F /= nil) orelse elixir_config:is_bootstrap() of true -> []; - false -> [["\n ", 'Elixir.Exception':format_stacktrace_entry(E)] || E <- Stacktrace] + false -> [["\n ", 'Elixir.Exception':format_stacktrace_entry(E)] || E <- ?key(Diagnostic, stacktrace)] end, io:put_chars(standard_error, [Output, MaybeStack, $\n, $\n]), @@ -139,12 +125,12 @@ extract_column(_) -> nil. %% Format snippets %% "Snippet" here refers to the source code line where the diagnostic/error occurred -format_snippet(_Position, nil, Message, nil, Severity, _Stacktrace, _Span) -> +format_snippet(Severity, _Position, nil, Message, nil, _Diagnostic) -> Formatted = [prefix(Severity), " ", Message], unicode:characters_to_binary(Formatted); -format_snippet(Position, File, Message, nil, Severity, Stacktrace, _Span) -> - Location = location_format(Position, File, Stacktrace), +format_snippet(Severity, Position, File, Message, nil, Diagnostic) -> + Location = location_format(Position, File, maps:get(stacktrace, Diagnostic, [])), Formatted = io_lib:format( "~ts ~ts\n" @@ -154,20 +140,22 @@ format_snippet(Position, File, Message, nil, Severity, Stacktrace, _Span) -> unicode:characters_to_binary(Formatted); -format_snippet(Position, File, Message, Snippet, Severity, Stacktrace, Span) -> +format_snippet(Severity, Position, File, Message, Snippet, Diagnostic) -> Column = extract_column(Position), LineNumber = extract_line(Position), LineDigits = get_line_number_digits(LineNumber, 1), Spacing = n_spaces(max(2, LineDigits) + 1), LineNumberSpacing = if LineDigits =:= 1 -> 1; true -> 0 end, {FormattedLine, ColumnsTrimmed} = format_line(Snippet), - Location = location_format(Position, File, Stacktrace), + Location = location_format(Position, File, maps:get(stacktrace, Diagnostic, [])), + MessageDetail = format_detail(Diagnostic, Message), Highlight = case Column of - nil -> highlight_below_line(FormattedLine, Severity); + nil -> + highlight_below_line(FormattedLine, Severity); _ -> - Length = calculate_span_length({LineNumber, Column}, Span), + Length = calculate_span_length({LineNumber, Column}, Diagnostic), highlight_at_position(Column - ColumnsTrimmed, Severity, Length) end, @@ -179,7 +167,7 @@ format_snippet(Position, File, Message, Snippet, Severity, Stacktrace, Span) -> " ~ts│\n" " ~ts└─ ~ts", [ - Spacing, prefix(Severity), format_message(Message, LineDigits, 2 + LineNumberSpacing), + Spacing, prefix(Severity), format_message(MessageDetail, LineDigits, 2 + LineNumberSpacing), Spacing, n_spaces(LineNumberSpacing), LineNumber, FormattedLine, Spacing, Highlight, @@ -189,9 +177,12 @@ format_snippet(Position, File, Message, Snippet, Severity, Stacktrace, Span) -> unicode:characters_to_binary(Formatted). -calculate_span_length({StartLine, StartCol}, {StartLine, EndCol}) -> EndCol - StartCol; -calculate_span_length({StartLine, _}, {EndLine, _}) when EndLine > StartLine -> 1; -calculate_span_length({_, _}, nil) -> 1. +format_detail(#{details := #{typing_traces := _}}, Message) -> [Message | "\ntyping violation found at:"]; +format_detail(_, Message) -> Message. + +calculate_span_length({StartLine, StartCol}, #{span := {StartLine, EndCol}}) -> EndCol - StartCol; +calculate_span_length({StartLine, _}, #{span := {EndLine, _}}) when EndLine > StartLine -> 1; +calculate_span_length({_, _}, #{}) -> 1. format_line(Line) -> case trim_line(Line, 0) of diff --git a/lib/elixir/src/elixir_expand.erl b/lib/elixir/src/elixir_expand.erl index 264793e9e5c..c10444643f9 100644 --- a/lib/elixir/src/elixir_expand.erl +++ b/lib/elixir/src/elixir_expand.erl @@ -340,7 +340,7 @@ expand({Name, Meta, Kind}, S, #{context := match} = E) when is_atom(Name), is_at %% Variable was already overridden #{Pair := VarVersion} when VarVersion >= PrematchVersion -> maybe_warn_underscored_var_repeat(Meta, Name, Kind, E), - NewUnused = var_used(Meta, Pair, VarVersion, Unused), + NewUnused = var_used(Pair, Meta, VarVersion, Unused), Var = {Name, [{version, VarVersion} | Meta], Kind}, {Var, S#elixir_ex{unused={NewUnused, Version}}, E}; @@ -396,7 +396,7 @@ expand({Name, Meta, Kind}, S, E) when is_atom(Name), is_atom(Kind) -> {ok, PairVersion} -> maybe_warn_underscored_var_access(Meta, Name, Kind, E), Var = {Name, [{version, PairVersion} | Meta], Kind}, - {Var, S#elixir_ex{unused={var_used(Meta, Pair, PairVersion, Unused), Version}}, E}; + {Var, S#elixir_ex{unused={var_used(Pair, Meta, PairVersion, Unused), Version}}, E}; Error -> case lists:keyfind(if_undefined, 1, Meta) of @@ -545,8 +545,6 @@ resolve_super(Meta, Arity, E) -> expand_fn_capture(Meta, Arg, S, E) -> case elixir_fn:capture(Meta, Arg, S, E) of {{remote, Remote, Fun, Arity}, RequireMeta, DotMeta, SE, EE} -> - is_atom(Remote) andalso - elixir_env:trace({remote_function, RequireMeta, Remote, Fun, Arity}, E), AttachedMeta = attach_runtime_module(Remote, RequireMeta, S, E), {{'&', Meta, [{'/', [], [{{'.', DotMeta, [Remote, Fun]}, AttachedMeta, []}, Arity]}]}, SE, EE}; {{local, Fun, Arity}, _, _, _SE, #{function := nil}} -> @@ -658,7 +656,7 @@ var_unused({_, Kind} = Pair, Meta, Version, Unused, Override) -> false -> Unused end. -var_used(Meta, {_, Kind} = Pair, Version, Unused) -> +var_used({_, Kind} = Pair, Meta, Version, Unused) -> KeepUnused = lists:keymember(keep_unused, 1, Meta), if diff --git a/lib/elixir/src/elixir_fn.erl b/lib/elixir/src/elixir_fn.erl index a7ef530830f..4dbb25983f2 100644 --- a/lib/elixir/src/elixir_fn.erl +++ b/lib/elixir/src/elixir_fn.erl @@ -128,12 +128,18 @@ validate(Meta, [{Pos, _} | _], Expected, E) -> validate(_Meta, [], _Pos, _E) -> []. -escape({'&', _, [Pos]}, _E, Dict) when is_integer(Pos), Pos > 0 -> +escape({'&', Meta, [Pos]}, E, Dict) when is_integer(Pos), Pos > 0 -> % Using a nil context here to emit warnings when variable is unused. % This might pollute user space but is unlikely because variables % named :"&1" are not valid syntax. - Var = {list_to_atom([$& | integer_to_list(Pos)]), [], nil}, - {Var, orddict:store(Pos, Var, Dict)}; + case orddict:find(Pos, Dict) of + {ok, Var} -> + {Var, Dict}; + error -> + Next = elixir_module:next_counter(?key(E, module)), + Var = {capture, [{counter, Next} | Meta], nil}, + {Var, orddict:store(Pos, Var, Dict)} + end; escape({'&', Meta, [Pos]}, E, _Dict) when is_integer(Pos) -> file_error(Meta, E, ?MODULE, {invalid_arity_for_capture, Pos}); escape({'&', Meta, _} = Arg, E, _Dict) -> diff --git a/lib/elixir/src/elixir_import.erl b/lib/elixir/src/elixir_import.erl index 853b81964e1..f985e232f23 100644 --- a/lib/elixir/src/elixir_import.erl +++ b/lib/elixir/src/elixir_import.erl @@ -2,12 +2,15 @@ %% between local functions and imports. %% For imports dispatch, please check elixir_dispatch. -module(elixir_import). --export([import/6, special_form/2, format_error/1]). +-export([import/6, import/7, special_form/2, format_error/1]). -compile(inline_list_funcs). -include("elixir.hrl"). import(Meta, Ref, Opts, E, Warn, Trace) -> - case import_only_except(Meta, Ref, Opts, E, Warn) of + import(Meta, Ref, Opts, E, Warn, Trace, fun Ref:'__info__'/1). + +import(Meta, Ref, Opts, E, Warn, Trace, InfoCallback) -> + case import_only_except(Meta, Ref, Opts, E, Warn, InfoCallback) of {Functions, Macros, Added} -> Trace andalso elixir_env:trace({import, [{imported, Added} | Meta], Ref, Opts}, E), EI = E#{functions := Functions, macros := Macros}, @@ -17,18 +20,18 @@ import(Meta, Ref, Opts, E, Warn, Trace) -> {error, Reason} end. -import_only_except(Meta, Ref, Opts, E, Warn) -> +import_only_except(Meta, Ref, Opts, E, Warn, InfoCallback) -> MaybeOnly = lists:keyfind(only, 1, Opts), case lists:keyfind(except, 1, Opts) of false -> - import_only_except(Meta, Ref, MaybeOnly, false, E, Warn); + import_only_except(Meta, Ref, MaybeOnly, false, E, Warn, InfoCallback); {except, DupExcept} when is_list(DupExcept) -> case ensure_keyword_list(DupExcept) of ok -> Except = ensure_no_duplicates(DupExcept, except, Meta, E, Warn), - import_only_except(Meta, Ref, MaybeOnly, Except, E, Warn); + import_only_except(Meta, Ref, MaybeOnly, Except, E, Warn, InfoCallback); error -> {error, {invalid_option, except, DupExcept}} @@ -38,27 +41,27 @@ import_only_except(Meta, Ref, Opts, E, Warn) -> {error, {invalid_option, except, Other}} end. -import_only_except(Meta, Ref, MaybeOnly, Except, E, Warn) -> +import_only_except(Meta, Ref, MaybeOnly, Except, E, Warn, InfoCallback) -> case MaybeOnly of {only, functions} -> - {Added1, _Used1, Funs} = import_functions(Meta, Ref, Except, E, Warn), + {Added1, _Used1, Funs} = import_functions(Meta, Ref, Except, E, Warn, InfoCallback), {Funs, keydelete(Ref, ?key(E, macros)), Added1}; {only, macros} -> - {Added2, _Used2, Macs} = import_macros(Meta, Ref, Except, E, Warn), + {Added2, _Used2, Macs} = import_macros(Meta, Ref, Except, E, Warn, InfoCallback), {keydelete(Ref, ?key(E, functions)), Macs, Added2}; {only, sigils} -> - {Added1, _Used1, Funs} = import_sigil_functions(Meta, Ref, Except, E, Warn), - {Added2, _Used2, Macs} = import_sigil_macros(Meta, Ref, Except, E, Warn), + {Added1, _Used1, Funs} = import_sigil_functions(Meta, Ref, Except, E, Warn, InfoCallback), + {Added2, _Used2, Macs} = import_sigil_macros(Meta, Ref, Except, E, Warn, InfoCallback), {Funs, Macs, Added1 or Added2}; {only, DupOnly} when is_list(DupOnly) -> case ensure_keyword_list(DupOnly) of ok when Except =:= false -> Only = ensure_no_duplicates(DupOnly, only, Meta, E, Warn), - {Added1, Used1, Funs} = import_listed_functions(Meta, Ref, Only, E, Warn), - {Added2, Used2, Macs} = import_listed_macros(Meta, Ref, Only, E, Warn), + {Added1, Used1, Funs} = import_listed_functions(Meta, Ref, Only, E, Warn, InfoCallback), + {Added2, Used2, Macs} = import_listed_macros(Meta, Ref, Only, E, Warn, InfoCallback), [Warn andalso elixir_errors:file_warn(Meta, E, ?MODULE, {invalid_import, {Ref, Name, Arity}}) || {Name, Arity} <- (Only -- Used1) -- Used2], {Funs, Macs, Added1 or Added2}; @@ -74,37 +77,37 @@ import_only_except(Meta, Ref, MaybeOnly, Except, E, Warn) -> {error, {invalid_option, only, Other}}; false -> - {Added1, _Used1, Funs} = import_functions(Meta, Ref, Except, E, Warn), - {Added2, _Used2, Macs} = import_macros(Meta, Ref, Except, E, Warn), + {Added1, _Used1, Funs} = import_functions(Meta, Ref, Except, E, Warn, InfoCallback), + {Added2, _Used2, Macs} = import_macros(Meta, Ref, Except, E, Warn, InfoCallback), {Funs, Macs, Added1 or Added2} end. -import_listed_functions(Meta, Ref, Only, E, Warn) -> - New = intersection(Only, get_functions(Ref)), +import_listed_functions(Meta, Ref, Only, E, Warn, InfoCallback) -> + New = intersection(Only, get_functions(Ref, InfoCallback)), calculate_key(Meta, Ref, ?key(E, functions), New, E, Warn). -import_listed_macros(Meta, Ref, Only, E, Warn) -> - New = intersection(Only, get_macros(Ref)), +import_listed_macros(Meta, Ref, Only, E, Warn, InfoCallback) -> + New = intersection(Only, get_macros(InfoCallback)), calculate_key(Meta, Ref, ?key(E, macros), New, E, Warn). -import_functions(Meta, Ref, Except, E, Warn) -> +import_functions(Meta, Ref, Except, E, Warn, InfoCallback) -> calculate_except(Meta, Ref, Except, ?key(E, functions), E, Warn, fun() -> - get_functions(Ref) + get_functions(Ref, InfoCallback) end). -import_macros(Meta, Ref, Except, E, Warn) -> +import_macros(Meta, Ref, Except, E, Warn, InfoCallback) -> calculate_except(Meta, Ref, Except, ?key(E, macros), E, Warn, fun() -> - get_macros(Ref) + get_macros(InfoCallback) end). -import_sigil_functions(Meta, Ref, Except, E, Warn) -> +import_sigil_functions(Meta, Ref, Except, E, Warn, InfoCallback) -> calculate_except(Meta, Ref, Except, ?key(E, functions), E, Warn, fun() -> - filter_sigils(get_functions(Ref)) + filter_sigils(InfoCallback(functions)) end). -import_sigil_macros(Meta, Ref, Except, E, Warn) -> +import_sigil_macros(Meta, Ref, Except, E, Warn, InfoCallback) -> calculate_except(Meta, Ref, Except, ?key(E, macros), E, Warn, fun() -> - filter_sigils(get_macros(Ref)) + filter_sigils(InfoCallback(macros)) end). calculate_except(Meta, Key, false, Old, E, Warn, Existing) -> @@ -134,16 +137,16 @@ calculate_key(Meta, Key, Old, New, E, Warn) -> %% Retrieve functions and macros from modules -get_functions(Module) -> +get_functions(Module, InfoCallback) -> try - Module:'__info__'(functions) + InfoCallback(functions) catch error:undef -> remove_internals(Module:module_info(exports)) end. -get_macros(Module) -> +get_macros(InfoCallback) -> try - Module:'__info__'(macros) + InfoCallback(macros) catch error:undef -> [] end. diff --git a/lib/elixir/test/elixir/calendar/duration_test.exs b/lib/elixir/test/elixir/calendar/duration_test.exs index ecf64f4e25c..1240d176aab 100644 --- a/lib/elixir/test/elixir/calendar/duration_test.exs +++ b/lib/elixir/test/elixir/calendar/duration_test.exs @@ -220,4 +220,142 @@ defmodule DurationTest do microsecond: {0, 0} } end + + test "from_iso8601/1" do + assert Duration.from_iso8601("P1Y2M3DT4H5M6S") == + {:ok, %Duration{year: 1, month: 2, day: 3, hour: 4, minute: 5, second: 6}} + + assert Duration.from_iso8601("P3WT5H3M") == {:ok, %Duration{week: 3, hour: 5, minute: 3}} + assert Duration.from_iso8601("PT5H3M") == {:ok, %Duration{hour: 5, minute: 3}} + assert Duration.from_iso8601("P1Y2M3D") == {:ok, %Duration{year: 1, month: 2, day: 3}} + assert Duration.from_iso8601("PT4H5M6S") == {:ok, %Duration{hour: 4, minute: 5, second: 6}} + assert Duration.from_iso8601("P1Y2M") == {:ok, %Duration{year: 1, month: 2}} + assert Duration.from_iso8601("P3D") == {:ok, %Duration{day: 3}} + assert Duration.from_iso8601("PT4H5M") == {:ok, %Duration{hour: 4, minute: 5}} + assert Duration.from_iso8601("PT6S") == {:ok, %Duration{second: 6}} + assert Duration.from_iso8601("P2M4Y") == {:error, :invalid_date_component} + assert Duration.from_iso8601("P4Y2W3Y") == {:error, :invalid_date_component} + assert Duration.from_iso8601("P5HT4MT3S") == {:error, :invalid_date_component} + assert Duration.from_iso8601("P5H3HT4M") == {:error, :invalid_date_component} + assert Duration.from_iso8601("P0.5Y") == {:error, :invalid_date_component} + assert Duration.from_iso8601("PT1D") == {:error, :invalid_time_component} + assert Duration.from_iso8601("PT.6S") == {:error, :invalid_time_component} + assert Duration.from_iso8601("PT0.5H") == {:error, :invalid_time_component} + assert Duration.from_iso8601("invalid") == {:error, :invalid_duration} + end + + test "from_iso8601!/1" do + assert Duration.from_iso8601!("P1Y2M3DT4H5M6S") == %Duration{ + year: 1, + month: 2, + day: 3, + hour: 4, + minute: 5, + second: 6 + } + + assert Duration.from_iso8601!("P3WT5H3M") == %Duration{week: 3, hour: 5, minute: 3} + assert Duration.from_iso8601!("PT5H3M") == %Duration{hour: 5, minute: 3} + assert Duration.from_iso8601!("P1Y2M3D") == %Duration{year: 1, month: 2, day: 3} + assert Duration.from_iso8601!("PT4H5M6S") == %Duration{hour: 4, minute: 5, second: 6} + assert Duration.from_iso8601!("P1Y2M") == %Duration{year: 1, month: 2} + assert Duration.from_iso8601!("P3D") == %Duration{day: 3} + assert Duration.from_iso8601!("PT4H5M") == %Duration{hour: 4, minute: 5} + assert Duration.from_iso8601!("PT6S") == %Duration{second: 6} + assert Duration.from_iso8601!("PT1,6S") == %Duration{second: 1, microsecond: {600_000, 1}} + assert Duration.from_iso8601!("PT-1.6S") == %Duration{second: -1, microsecond: {-600_000, 1}} + assert Duration.from_iso8601!("PT0,6S") == %Duration{second: 0, microsecond: {600_000, 1}} + assert Duration.from_iso8601!("PT-0,6S") == %Duration{second: 0, microsecond: {-600_000, 1}} + assert Duration.from_iso8601!("-PT-0,6S") == %Duration{second: 0, microsecond: {600_000, 1}} + assert Duration.from_iso8601!("-P10DT4H") == %Duration{day: -10, hour: -4} + assert Duration.from_iso8601!("-P10DT-4H") == %Duration{day: -10, hour: 4} + assert Duration.from_iso8601!("P-10D") == %Duration{day: -10} + assert Duration.from_iso8601!("+P10DT-4H") == %Duration{day: 10, hour: -4} + assert Duration.from_iso8601!("P+10D") == %Duration{day: 10} + assert Duration.from_iso8601!("-P+10D") == %Duration{day: -10} + + assert Duration.from_iso8601!("PT-1.234567S") == %Duration{ + second: -1, + microsecond: {-234_567, 6} + } + + assert Duration.from_iso8601!("PT1.12345678S") == %Duration{ + second: 1, + microsecond: {123_456, 6} + } + + assert Duration.from_iso8601!("P3Y4W-3DT-6S") == %Duration{ + year: 3, + week: 4, + day: -3, + second: -6 + } + + assert Duration.from_iso8601!("PT-4.23S") == %Duration{second: -4, microsecond: {-230_000, 2}} + + assert_raise ArgumentError, + ~s/failed to parse duration "P5H3HT4M". reason: :invalid_date_component/, + fn -> + Duration.from_iso8601!("P5H3HT4M") + end + + assert_raise ArgumentError, + ~s/failed to parse duration "P4Y2W3Y". reason: :invalid_date_component/, + fn -> + Duration.from_iso8601!("P4Y2W3Y") + end + + assert_raise ArgumentError, + ~s/failed to parse duration "invalid". reason: :invalid_duration/, + fn -> + Duration.from_iso8601!("invalid") + end + + assert_raise ArgumentError, + ~s/failed to parse duration "P4.5YT6S". reason: :invalid_date_component/, + fn -> + Duration.from_iso8601!("P4.5YT6S") + end + end + + test "to_iso8601/1" do + assert %Duration{year: 1, month: 2, day: 3, hour: 4, minute: 5, second: 6} + |> Duration.to_iso8601() == "P1Y2M3DT4H5M6S" + + assert %Duration{week: 3, hour: 5, minute: 3} |> Duration.to_iso8601() == "P3WT5H3M" + assert %Duration{hour: 5, minute: 3} |> Duration.to_iso8601() == "PT5H3M" + assert %Duration{year: 1, month: 2, day: 3} |> Duration.to_iso8601() == "P1Y2M3D" + assert %Duration{hour: 4, minute: 5, second: 6} |> Duration.to_iso8601() == "PT4H5M6S" + assert %Duration{year: 1, month: 2} |> Duration.to_iso8601() == "P1Y2M" + assert %Duration{day: 3} |> Duration.to_iso8601() == "P3D" + assert %Duration{hour: 4, minute: 5} |> Duration.to_iso8601() == "PT4H5M" + assert %Duration{second: 6} |> Duration.to_iso8601() == "PT6S" + assert %Duration{second: 1, microsecond: {600_000, 1}} |> Duration.to_iso8601() == "PT1.6S" + assert %Duration{second: -1, microsecond: {-600_000, 1}} |> Duration.to_iso8601() == "PT-1.6S" + + assert %Duration{second: -1, microsecond: {-234_567, 6}} |> Duration.to_iso8601() == + "PT-1.234567S" + + assert %Duration{second: 1, microsecond: {123_456, 6}} |> Duration.to_iso8601() == + "PT1.123456S" + + assert %Duration{year: 3, week: 4, day: -3, second: -6} |> Duration.to_iso8601() == + "P3Y4W-3DT-6S" + + assert %Duration{second: -4, microsecond: {-230_000, 2}} |> Duration.to_iso8601() == + "PT-4.23S" + + assert %Duration{second: -4, microsecond: {230_000, 2}} |> Duration.to_iso8601() == + "PT-3.77S" + + assert %Duration{second: 2, microsecond: {-1_200_000, 4}} |> Duration.to_iso8601() == + "PT0.8000S" + + assert %Duration{second: 1, microsecond: {-1_200_000, 3}} |> Duration.to_iso8601() == + "PT-0.200S" + + assert %Duration{microsecond: {-800_000, 2}} |> Duration.to_iso8601() == "PT-0.80S" + assert %Duration{microsecond: {-800_000, 0}} |> Duration.to_iso8601() == "PT0S" + assert %Duration{microsecond: {-1_200_000, 2}} |> Duration.to_iso8601() == "PT-1.20S" + end end diff --git a/lib/elixir/test/elixir/code_normalizer/quoted_ast_test.exs b/lib/elixir/test/elixir/code_normalizer/quoted_ast_test.exs index 387e109e48d..ac392c26adb 100644 --- a/lib/elixir/test/elixir/code_normalizer/quoted_ast_test.exs +++ b/lib/elixir/test/elixir/code_normalizer/quoted_ast_test.exs @@ -172,6 +172,20 @@ defmodule Code.Normalizer.QuotedASTTest do ) == ~s[~S"""\n"123"\n"""] end + test "regression: invalid sigil calls" do + assert quoted_to_string(quote do: sigil_r(<<"foo", 123>>, [])) == + "sigil_r(<<\"foo\", 123>>, [])" + + assert quoted_to_string(quote do: sigil_r(<<"foo">>, :invalid_modifiers)) == + "sigil_r(\"foo\", :invalid_modifiers)" + + assert quoted_to_string(quote do: sigil_r(<<"foo">>, [:invalid_modifier])) == + "sigil_r(\"foo\", [:invalid_modifier])" + + assert quoted_to_string(quote do: sigil_r(<<"foo">>, [])) == "~r\"foo\"" + assert quoted_to_string(quote do: sigil_r(<<"foo">>, [?a, ?b, ?c])) == "~r\"foo\"abc" + end + test "tuple" do assert quoted_to_string(quote do: {1, 2}) == "{1, 2}" assert quoted_to_string(quote do: {1}) == "{1}" diff --git a/lib/elixir/test/elixir/fixtures/dialyzer/with_no_return.ex b/lib/elixir/test/elixir/fixtures/dialyzer/with_no_return.ex new file mode 100644 index 00000000000..264f58c8ff8 --- /dev/null +++ b/lib/elixir/test/elixir/fixtures/dialyzer/with_no_return.ex @@ -0,0 +1,13 @@ +defmodule Dialyzer.WithNoReturn do + def with_no_return(list) do + no_return = fn -> throw(:no_return) end + + with [] <- list do + :ok + else + # note: throwing here directly wouldn't be caught in the first place, + # calling a no_return function is what could cause an issue. + _ -> no_return.() + end + end +end diff --git a/lib/elixir/test/elixir/fixtures/dialyzer/with_throwing_else.ex b/lib/elixir/test/elixir/fixtures/dialyzer/with_throwing_else.ex new file mode 100644 index 00000000000..1ea99134e37 --- /dev/null +++ b/lib/elixir/test/elixir/fixtures/dialyzer/with_throwing_else.ex @@ -0,0 +1,12 @@ +defmodule Dialyzer.WithThrowingElse do + def with_throwing_else(map) do + with {:ok, foo} <- Map.fetch(map, :foo), + false <- Enum.empty?(foo) do + foo + else + # several clauses but one is a no_return + :error -> throw(:empty_map) + true -> nil + end + end +end diff --git a/lib/elixir/test/elixir/kernel/dialyzer_test.exs b/lib/elixir/test/elixir/kernel/dialyzer_test.exs index 889ec4e4ef9..164f09e5ee7 100644 --- a/lib/elixir/test/elixir/kernel/dialyzer_test.exs +++ b/lib/elixir/test/elixir/kernel/dialyzer_test.exs @@ -26,6 +26,7 @@ defmodule Kernel.DialyzerTest do :elixir, :elixir_env, :elixir_erl_pass, + :maps, ArgumentError, Atom, Code, @@ -159,6 +160,16 @@ defmodule Kernel.DialyzerTest do assert_dialyze_no_warnings!(context) end + test "no warnings on with when else has a no_return type", context do + copy_beam!(context, Dialyzer.WithNoReturn) + assert_dialyze_no_warnings!(context) + end + + test "no warnings on with when multiple else clauses and one is a no_return", context do + copy_beam!(context, Dialyzer.WithThrowingElse) + assert_dialyze_no_warnings!(context) + end + test "no warnings on defmacrop", context do copy_beam!(context, Dialyzer.Defmacrop) assert_dialyze_no_warnings!(context) diff --git a/lib/elixir/test/elixir/kernel/errors_test.exs b/lib/elixir/test/elixir/kernel/errors_test.exs index 5e526bb730a..fdb658c044b 100644 --- a/lib/elixir/test/elixir/kernel/errors_test.exs +++ b/lib/elixir/test/elixir/kernel/errors_test.exs @@ -634,6 +634,16 @@ defmodule Kernel.ErrorsTest do end """ ) + + assert_compile_error( + ["nofile:3:17", "function exit/1 imported from both :erlang and Kernel, call is ambiguous"], + ~c""" + defmodule Kernel.ErrorsTest.FunctionImportConflict do + import :erlang, only: [exit: 1], warn: false + def foo, do: &exit/1 + end + """ + ) end test "ensure valid import :only option" do diff --git a/lib/elixir/test/elixir/kernel/expansion_test.exs b/lib/elixir/test/elixir/kernel/expansion_test.exs index 629dde62aaf..8a751fe1a45 100644 --- a/lib/elixir/test/elixir/kernel/expansion_test.exs +++ b/lib/elixir/test/elixir/kernel/expansion_test.exs @@ -621,23 +621,6 @@ defmodule Kernel.ExpansionTest do assert expand(quote(do: quote(do: hello)), []) == {:{}, [], [:hello, [], __MODULE__]} end - test "expand bind_quoted once" do - expand_env( - quote do - var = 123 - quote(bind_quoted: [var: var], do: var) - end, - __ENV__, - [] - ) - |> elem(0) - |> Macro.prewalk(fn - {:var, [version: 0], Kernel.ExpansionTest} -> :ok - {:var, _, Kernel.ExpansionTest} = invalid -> flunk("unexpected node #{inspect(invalid)}") - node -> node - end) - end - test "raises if the :bind_quoted option is invalid" do assert_compile_error(~r"invalid :bind_quoted for quote", fn -> expand(quote(do: quote(bind_quoted: self(), do: :ok))) @@ -1150,6 +1133,12 @@ defmodule Kernel.ExpansionTest do assert expand(quote(do: &unknown(&1, &2))) == {:&, [], [{:/, [], [{:unknown, [], nil}, 2]}]} end + test "keeps position meta on & variables" do + assert expand(Code.string_to_quoted!("& &1")) |> clean_meta([:counter]) == + {:fn, [{:line, 1}], + [{:->, [{:line, 1}], [[{:capture, [line: 1], nil}], {:capture, [line: 1], nil}]}]} + end + test "expands remotes" do assert expand(quote(do: &List.flatten/2)) == quote(do: &:"Elixir.List".flatten/2) diff --git a/lib/elixir/test/elixir/kernel/fn_test.exs b/lib/elixir/test/elixir/kernel/fn_test.exs index b1f8f4b4267..1304d753577 100644 --- a/lib/elixir/test/elixir/kernel/fn_test.exs +++ b/lib/elixir/test/elixir/kernel/fn_test.exs @@ -88,10 +88,6 @@ defmodule Kernel.FnTest do assert is_function(&and/2) end - test "capture precedence in cons" do - assert [(&IO.puts/1) | &IO.puts/2] == [(&IO.puts/1) | &IO.puts/2] - end - test "capture with variable module" do mod = List assert (&mod.flatten(&1)).([1, [2], 3]) == [1, 2, 3] @@ -140,13 +136,23 @@ defmodule Kernel.FnTest do assert (&(!is_atom(&1))).(:foo) == false end - test "capture other" do + test "capture with function call" do assert (& &1).(:ok) == :ok fun = fn a, b -> a + b end assert (&fun.(&1, 2)).(1) == 3 end + defmacro c(x) do + quote do + &(unquote(x) <> &1) + end + end + + test "capture within capture through macro" do + assert (&c(&1).("b")).("a") == "ab" + end + defp atom?(atom) when is_atom(atom), do: true defp atom?(_), do: false diff --git a/lib/elixir/test/elixir/kernel/tracers_test.exs b/lib/elixir/test/elixir/kernel/tracers_test.exs index ee282c10261..fc788070754 100644 --- a/lib/elixir/test/elixir/kernel/tracers_test.exs +++ b/lib/elixir/test/elixir/kernel/tracers_test.exs @@ -27,8 +27,8 @@ defmodule Kernel.TracersTest do Foo """) - assert_receive {:start, %{lexical_tracker: pid}} when is_pid(pid) - assert_receive {:stop, %{lexical_tracker: pid}} when is_pid(pid) + assert_received {:start, %{lexical_tracker: pid}} when is_pid(pid) + assert_received {:stop, %{lexical_tracker: pid}} when is_pid(pid) end test "traces alias references" do @@ -36,7 +36,7 @@ defmodule Kernel.TracersTest do Foo """) - assert_receive {{:alias_reference, meta, Foo}, _} + assert_received {{:alias_reference, meta, Foo}, _} assert meta[:line] == 1 assert meta[:column] == 1 end @@ -50,17 +50,17 @@ defmodule Kernel.TracersTest do Bar """) - assert_receive {{:alias, meta, Hello.World, World, []}, _} + assert_received {{:alias, meta, Hello.World, World, []}, _} assert meta[:line] == 1 assert meta[:column] == 1 - assert_receive {{:alias_expansion, meta, World, Hello.World}, _} + assert_received {{:alias_expansion, meta, World, Hello.World}, _} assert meta[:line] == 2 assert meta[:column] == 1 - assert_receive {{:alias, meta, Foo, Bar, [as: Bar, warn: true]}, _} + assert_received {{:alias, meta, Foo, Bar, [as: Bar, warn: true]}, _} assert meta[:line] == 4 assert meta[:column] == 1 - assert_receive {{:alias_expansion, meta, Bar, Foo}, _} + assert_received {{:alias_expansion, meta, Bar, Foo}, _} assert meta[:line] == 5 assert meta[:column] == 1 end @@ -72,15 +72,19 @@ defmodule Kernel.TracersTest do {1, ""} = parse("1") """) - assert_receive {{:import, meta, Integer, only: [is_odd: 1, parse: 1]}, _} + assert_received {{:import, meta, Integer, only: [is_odd: 1, parse: 1]}, _} assert meta[:line] == 1 assert meta[:column] == 1 - assert_receive {{:imported_macro, meta, Integer, :is_odd, 1}, _} + assert_received {{:imported_macro, meta, Integer, :is_odd, 1}, _} assert meta[:line] == 2 assert meta[:column] == 8 - assert_receive {{:imported_function, meta, Integer, :parse, 1}, _} + assert_received {{:imported_function, meta, Integer, :parse, 1}, _} + assert meta[:line] == 3 + assert meta[:column] == 11 + + assert_received {{:remote_function, meta, Integer, :parse, 1}, _} assert meta[:line] == 3 assert meta[:column] == 11 end @@ -92,15 +96,15 @@ defmodule Kernel.TracersTest do &parse/1 """) - assert_receive {{:import, meta, Integer, only: [is_odd: 1, parse: 1]}, _} + assert_received {{:import, meta, Integer, only: [is_odd: 1, parse: 1]}, _} assert meta[:line] == 1 assert meta[:column] == 1 - assert_receive {{:imported_macro, meta, Integer, :is_odd, 1}, _} + assert_received {{:imported_macro, meta, Integer, :is_odd, 1}, _} assert meta[:line] == 2 assert meta[:column] == 2 - assert_receive {{:imported_function, meta, Integer, :parse, 1}, _} + assert_received {{:imported_function, meta, Integer, :parse, 1}, _} assert meta[:line] == 3 assert meta[:column] == 2 end @@ -110,7 +114,7 @@ defmodule Kernel.TracersTest do %URI{path: "/"} """) - assert_receive {{:struct_expansion, meta, URI, [:path]}, _} + assert_received {{:struct_expansion, meta, URI, [:path]}, _} assert meta[:line] == 1 assert meta[:column] == 1 end @@ -123,15 +127,15 @@ defmodule Kernel.TracersTest do "foo" = Atom.to_string(:foo) """) - assert_receive {{:remote_macro, meta, Integer, :is_odd, 1}, _} + assert_received {{:remote_macro, meta, Integer, :is_odd, 1}, _} assert meta[:line] == 2 assert meta[:column] == 16 - assert_receive {{:remote_function, meta, Integer, :parse, 1}, _} + assert_received {{:remote_function, meta, Integer, :parse, 1}, _} assert meta[:line] == 3 assert meta[:column] == 19 - assert_receive {{:remote_function, meta, Atom, :to_string, 1}, _} + assert_received {{:remote_function, meta, Atom, :to_string, 1}, _} assert meta[:line] == 4 assert meta[:column] == 14 end @@ -143,11 +147,11 @@ defmodule Kernel.TracersTest do &Integer.parse/1 """) - assert_receive {{:remote_macro, meta, Integer, :is_odd, 1}, _} + assert_received {{:remote_macro, meta, Integer, :is_odd, 1}, _} assert meta[:line] == 2 assert meta[:column] == 10 - assert_receive {{:remote_function, meta, Integer, :parse, 1}, _} + assert_received {{:remote_function, meta, Integer, :parse, 1}, _} assert meta[:line] == 3 assert meta[:column] == 10 end @@ -161,11 +165,11 @@ defmodule Kernel.TracersTest do end """) - assert_receive {{:local_macro, meta, :foo, 1}, _} + assert_received {{:local_macro, meta, :foo, 1}, _} assert meta[:line] == 4 assert meta[:column] == 21 - assert_receive {{:local_function, meta, :bar, 1}, _} + assert_received {{:local_function, meta, :bar, 1}, _} assert meta[:line] == 4 assert meta[:column] == 32 after @@ -182,11 +186,11 @@ defmodule Kernel.TracersTest do end """) - assert_receive {{:local_macro, meta, :foo, 1}, _} + assert_received {{:local_macro, meta, :foo, 1}, _} assert meta[:line] == 4 assert meta[:column] == 21 - assert_receive {{:local_function, meta, :bar, 1}, _} + assert_received {{:local_function, meta, :bar, 1}, _} assert meta[:line] == 4 assert meta[:column] == 29 after @@ -201,8 +205,8 @@ defmodule Kernel.TracersTest do end """) - assert_receive {:defmodule, %{module: Sample, function: nil}} - assert_receive {{:on_module, <<_::binary>>, :none}, %{module: Sample, function: nil}} + assert_received {:defmodule, %{module: Sample, function: nil}} + assert_received {{:on_module, <<_::binary>>, :none}, %{module: Sample, function: nil}} after :code.purge(Sample) :code.delete(Sample) @@ -213,8 +217,8 @@ defmodule Kernel.TracersTest do Module.create(Sample, :ok, __ENV__) """) - assert_receive {:defmodule, %{module: Sample, function: nil}} - assert_receive {{:on_module, <<_::binary>>, :none}, %{module: Sample, function: nil}} + assert_received {:defmodule, %{module: Sample, function: nil}} + assert_received {{:on_module, <<_::binary>>, :none}, %{module: Sample, function: nil}} after :code.purge(Sample) :code.delete(Sample) @@ -226,7 +230,7 @@ defmodule Kernel.TracersTest do "foo\#{arg}" """) - assert_receive {{:remote_macro, meta, Kernel, :to_string, 1}, _env} + assert_received {{:remote_macro, meta, Kernel, :to_string, 1}, _env} assert meta[:from_interpolation] end @@ -236,7 +240,7 @@ defmodule Kernel.TracersTest do foo[:bar] """) - assert_receive {{:remote_function, meta, Access, :get, 2}, _env} + assert_received {{:remote_function, meta, Access, :get, 2}, _env} assert meta[:from_brackets] compile_string(""" @@ -248,17 +252,37 @@ defmodule Kernel.TracersTest do end """) - assert_receive {{:remote_function, meta, Access, :get, 2}, _env} + assert_received {{:remote_function, meta, Access, :get, 2}, _env} assert meta[:from_brackets] compile_string(""" %{bar: 3}[:bar] """) - assert_receive {{:remote_function, meta, Access, :get, 2}, _env} + assert_received {{:remote_function, meta, Access, :get, 2}, _env} assert meta[:from_brackets] end + test "does not trace bind quoted twice" do + compile_string(""" + quote bind_quoted: [foo: List.flatten([])] do + foo + end + """) + + assert_received {{:remote_function, _, List, :flatten, 1}, _} + refute_received {{:remote_function, _, List, :flatten, 1}, _} + end + + test "does not trace captures twice" do + compile_string(""" + &List.flatten/1 + """) + + assert_received {{:remote_function, _, List, :flatten, 1}, _} + refute_received {{:remote_function, _, List, :flatten, 1}, _} + end + """ # Make sure this module is compiled with column information defmodule MacroWithColumn do @@ -278,7 +302,7 @@ defmodule Kernel.TracersTest do MacroWithColumn.some_macro(["hello", "world", "!"]) """) - assert_receive {{:alias_reference, meta, Enum}, _env} + assert_received {{:alias_reference, meta, Enum}, _env} refute meta[:column] end end diff --git a/lib/elixir/test/elixir/kernel/warning_test.exs b/lib/elixir/test/elixir/kernel/warning_test.exs index bf656cea5b9..a604a0f530c 100644 --- a/lib/elixir/test/elixir/kernel/warning_test.exs +++ b/lib/elixir/test/elixir/kernel/warning_test.exs @@ -463,17 +463,6 @@ defmodule Kernel.WarningTest do fn x -> match?(x, :value) end """ ) - - assert_warn_eval( - [ - "nofile:1", - "variable \"&1\" is unused (this might happen when using a capture argument as a pattern)", - "variable \"&1\" is unused (this might happen when using a capture argument as a pattern)" - ], - """ - &match?(&1, :value) - """ - ) end test "useless literal" do diff --git a/lib/elixir/test/elixir/kernel/with_test.exs b/lib/elixir/test/elixir/kernel/with_test.exs index 0ad008186ba..c4a17f18e67 100644 --- a/lib/elixir/test/elixir/kernel/with_test.exs +++ b/lib/elixir/test/elixir/kernel/with_test.exs @@ -25,6 +25,30 @@ defmodule Kernel.WithTest do assert with({^key, res} <- ok(42), do: res) == 42 end + test "pin matching with multiple else" do + key = :error + + first_else = + with nil <- error() do + :ok + else + ^key -> :pinned + _other -> :other + end + + assert first_else == :pinned + + second_else = + with nil <- ok(42) do + :ok + else + ^key -> :pinned + _other -> :other + end + + assert second_else == :other + end + test "two levels with" do result = with {:ok, n1} <- ok(11), diff --git a/lib/elixir/test/elixir/keyword_test.exs b/lib/elixir/test/elixir/keyword_test.exs index a80d152885a..f4791aa709d 100644 --- a/lib/elixir/test/elixir/keyword_test.exs +++ b/lib/elixir/test/elixir/keyword_test.exs @@ -21,11 +21,7 @@ defmodule KeywordTest do end test "implements (almost) all functions in Map" do - assert Map.__info__(:functions) -- Keyword.__info__(:functions) == [ - from_struct: 1, - intersect: 2, - intersect: 3 - ] + assert Map.__info__(:functions) -- Keyword.__info__(:functions) == [from_struct: 1] end test "get_and_update/3 raises on bad return value from the argument function" do diff --git a/lib/elixir/test/elixir/macro_test.exs b/lib/elixir/test/elixir/macro_test.exs index a1f697685a8..967846c1ffd 100644 --- a/lib/elixir/test/elixir/macro_test.exs +++ b/lib/elixir/test/elixir/macro_test.exs @@ -190,6 +190,16 @@ defmodule MacroTest do assert Code.eval_quoted(expanded) == {env.versioned_vars, []} end + test "env in :match context does not expand" do + env = %{__ENV__ | line: 0, lexical_tracker: self(), context: :match} + + expanded = Macro.expand_once(quote(do: __ENV__), env) + assert expanded == quote(do: __ENV__) + + expanded = Macro.expand_once(quote(do: __ENV__.file), env) + assert expanded == quote(do: __ENV__.file) + end + defmacro local_macro(), do: raise("ignored") test "vars" do diff --git a/lib/elixir/test/elixir/module/types/descr_test.exs b/lib/elixir/test/elixir/module/types/descr_test.exs index 7260b46e172..98acbc79822 100644 --- a/lib/elixir/test/elixir/module/types/descr_test.exs +++ b/lib/elixir/test/elixir/module/types/descr_test.exs @@ -13,6 +13,7 @@ defmodule Module.Types.DescrTest do test "term" do assert union(term(), float()) == term() assert union(term(), binary()) == term() + assert union(term(), if_set(binary())) == if_set(term()) end test "none" do @@ -45,7 +46,7 @@ defmodule Module.Types.DescrTest do reference() ] - assert Enum.reduce(all, &union/2) == term() + assert Enum.reduce(all, &union/2) |> equal?(term()) end test "dynamic" do @@ -214,6 +215,8 @@ defmodule Module.Types.DescrTest do # optional refute subtype?(closed_map(a: if_set(integer())), closed_map(a: integer())) assert subtype?(closed_map(a: integer()), closed_map(a: if_set(integer()))) + refute subtype?(closed_map(a: if_set(term())), closed_map(a: term())) + assert subtype?(closed_map(a: term()), closed_map(a: if_set(term()))) end end @@ -253,12 +256,34 @@ defmodule Module.Types.DescrTest do end end + describe "queries" do + test "atom_type?" do + assert atom_type?(term(), :foo) + assert atom_type?(dynamic(), :foo) + + assert atom_type?(atom([:foo, :bar]), :foo) + refute atom_type?(atom([:foo, :bar]), :baz) + assert atom_type?(negation(atom([:foo, :bar])), :baz) + + refute atom_type?(union(atom([:foo, :bar]), integer()), :baz) + refute atom_type?(dynamic(union(atom([:foo, :bar]), integer())), :baz) + end + end + describe "projections" do + test "fun_fetch" do + assert fun_fetch(term(), 1) == :error + assert fun_fetch(union(term(), dynamic(fun())), 1) == :error + assert fun_fetch(fun(), 1) == :ok + assert fun_fetch(dynamic(), 1) == :ok + end + test "atom_fetch" do assert atom_fetch(term()) == :error assert atom_fetch(union(term(), dynamic(atom([:foo, :bar])))) == :error assert atom_fetch(atom()) == {:infinite, []} + assert atom_fetch(dynamic()) == {:infinite, []} assert atom_fetch(atom([:foo, :bar])) == {:finite, [:foo, :bar] |> :sets.from_list(version: 2) |> :sets.to_list()} diff --git a/lib/elixir/test/elixir/module/types/expr_test.exs b/lib/elixir/test/elixir/module/types/expr_test.exs index d67f820fa45..64123aabe20 100644 --- a/lib/elixir/test/elixir/module/types/expr_test.exs +++ b/lib/elixir/test/elixir/module/types/expr_test.exs @@ -21,6 +21,32 @@ defmodule Module.Types.ExprTest do assert typecheck!(fn -> :ok end) == fun() end + describe "funs" do + test "incompatible" do + assert typewarn!([%x{}], x.(1, 2)) == + {dynamic(), + ~l""" + incompatible types in expression: + + x + + expected type: + + fun() + + but got type: + + dynamic(atom()) + + where "x" was given the type: + + # type: dynamic(atom()) + # from: types_test.ex:LINE-2 + %x{} + """} + end + end + describe "remotes" do test "dynamic calls" do assert typecheck!([%x{}], x.foo_bar()) == dynamic() @@ -52,8 +78,6 @@ defmodule Module.Types.ExprTest do <> #{hints(:dot)} - - typing violation found at:\ """} end @@ -70,8 +94,6 @@ defmodule Module.Types.ExprTest do # type: integer() # from: types_test.ex:LINE-2 <> - - typing violation found at:\ """} end @@ -92,8 +114,6 @@ defmodule Module.Types.ExprTest do # type: integer() # from: types_test.ex:LINE-2 <> - - typing violation found at:\ """} end end @@ -120,8 +140,6 @@ defmodule Module.Types.ExprTest do # type: binary() # from: types_test.ex:LINE-2 <> - - typing violation found at:\ """} assert typewarn!([<>], <>) == @@ -146,8 +164,6 @@ defmodule Module.Types.ExprTest do <> #{hints(:inferred_bitstring_spec)} - - typing violation found at:\ """} assert typewarn!([<>], <>) == @@ -172,8 +188,6 @@ defmodule Module.Types.ExprTest do <> #{hints(:inferred_bitstring_spec)} - - typing violation found at:\ """} end end @@ -240,8 +254,6 @@ defmodule Module.Types.ExprTest do # type: :foo # from: types_test.ex:LINE-2 x = :foo - - typing violation found at:\ """} end @@ -264,52 +276,52 @@ defmodule Module.Types.ExprTest do <> #{hints(:dot)} - - typing violation found at:\ """} end - test "accessing an unknown field on struct" do - assert typewarn!(%Point{}.foo_bar) == - {dynamic(), - ~l""" - unknown key .foo_bar in expression: + test "accessing an unknown field on struct with diagnostic" do + {type, diagnostic} = typediag!(%Point{}.foo_bar) + assert type == dynamic() + assert diagnostic.span == {__ENV__.line - 2, 54} - %Point{x: nil, y: nil, z: 0}.foo_bar + assert diagnostic.message == ~l""" + unknown key .foo_bar in expression: - the given type does not have the given key: + %Point{x: nil, y: nil, z: 0}.foo_bar - dynamic(%Point{x: nil, y: nil, z: integer()}) + the given type does not have the given key: - typing violation found at:\ - """} + dynamic(%Point{x: nil, y: nil, z: integer()}) + """ end - test "accessing an unknown field on struct in a var" do - assert typewarn!([x = %URI{}], x.foo_bar) == - {dynamic(), - ~l""" - unknown key .foo_bar in expression: - - x.foo_bar - - where "x" was given the type: - - # type: dynamic(%URI{ - authority: term(), - fragment: term(), - host: term(), - path: term(), - port: term(), - query: term(), - scheme: term(), - userinfo: term() - }) - # from: types_test.ex:LINE-2 - x = %URI{} - - typing violation found at:\ - """} + test "accessing an unknown field on struct in a var with diagnostic" do + {type, diagnostic} = typediag!([x = %URI{}], x.foo_bar) + assert type == dynamic() + assert diagnostic.span == {__ENV__.line - 2, 61} + + assert diagnostic.message == ~l""" + unknown key .foo_bar in expression: + + x.foo_bar + + where "x" was given the type: + + # type: dynamic(%URI{ + authority: term(), + fragment: term(), + host: term(), + path: term(), + port: term(), + query: term(), + scheme: term(), + userinfo: term() + }) + # from: types_test.ex:LINE-4 + x = %URI{} + """ + + assert [%{type: :variable, name: :x}] = diagnostic.details.typing_traces end end @@ -341,8 +353,6 @@ defmodule Module.Types.ExprTest do While Elixir can compare across all types, you are comparing across types \ which are always distinct, and the result is either always true or always false - - typing violation found at:\ """} end @@ -372,9 +382,7 @@ defmodule Module.Types.ExprTest do # from: types_test.ex:LINE-2 y = %Point{} - Comparison operators (>, <, >=, <=, min, and max) perform structural and not semantic comparison. Comparing with a struct won't give meaningful results. Struct that can be compared typically define a compare/2 function within their modules that can be used for semantic comparison - - typing violation found at:\ + Comparison operators (>, <, >=, <=, min, and max) perform structural and not semantic comparison. Comparing with a struct won't give meaningful results. Structs that can be compared typically define a compare/2 function within their modules that can be used for semantic comparison. """} end end @@ -433,8 +441,6 @@ defmodule Module.Types.ExprTest do ) # from: types_test.ex:LINE-5 rescue e in [SyntaxError, RuntimeError] -> - - typing violation found at:\ """} end @@ -460,8 +466,6 @@ defmodule Module.Types.ExprTest do rescue e -> #{hints(:anonymous_rescue)} - - typing violation found at:\ """} end end diff --git a/lib/elixir/test/elixir/module/types/pattern_test.exs b/lib/elixir/test/elixir/module/types/pattern_test.exs index 46597fba7aa..ad4f722260f 100644 --- a/lib/elixir/test/elixir/module/types/pattern_test.exs +++ b/lib/elixir/test/elixir/module/types/pattern_test.exs @@ -56,8 +56,6 @@ defmodule Module.Types.PatternTest do # type: integer() # from: types_test.ex:LINE-1 m = 123 - - typing violation found at:\ """ end @@ -74,8 +72,6 @@ defmodule Module.Types.PatternTest do # type: dynamic(%Point{x: term(), y: term(), z: term()}) # from: types_test.ex:LINE-2 x = %Point{} - - typing violation found at:\ """} end end @@ -115,8 +111,6 @@ defmodule Module.Types.PatternTest do # type: float() # from: types_test.ex:LINE <<..., x::float>> - - typing violation found at:\ """ assert typeerror!([<>], x) == ~l""" @@ -135,8 +129,6 @@ defmodule Module.Types.PatternTest do <<..., x>> #{hints(:inferred_bitstring_spec)} - - typing violation found at:\ """ end end diff --git a/lib/elixir/test/elixir/module/types/type_helper.exs b/lib/elixir/test/elixir/module/types/type_helper.exs index f9a58c2fb11..85fc2c20685 100644 --- a/lib/elixir/test/elixir/module/types/type_helper.exs +++ b/lib/elixir/test/elixir/module/types/type_helper.exs @@ -38,6 +38,16 @@ defmodule TypeHelper do end end + @doc """ + Main helper for checking the diagnostic of a given AST. + """ + defmacro typediag!(patterns \\ [], guards \\ [], body) do + quote do + unquote(typecheck(patterns, guards, body, __CALLER__)) + |> TypeHelper.__typediag__!() + end + end + @doc false def __typecheck__!({:ok, type, %{warnings: []}}), do: type @@ -49,24 +59,30 @@ defmodule TypeHelper do @doc false def __typeerror__!({:error, %{warnings: [{module, warning, _locs} | _]}}), - do: warning |> module.format_warning() |> IO.iodata_to_binary() + do: module.format_diagnostic(warning).message def __typeerror__!({:ok, type, _context}), do: raise("type checking ok but expected error: #{Descr.to_quoted_string(type)}") @doc false - def __typewarn__!({:ok, type, %{warnings: [{module, warning, _locs}]}}), - do: {type, warning |> module.format_warning() |> IO.iodata_to_binary()} + def __typediag__!({:ok, type, %{warnings: [{module, warning, _locs}]}}), + do: {type, module.format_diagnostic(warning)} - def __typewarn__!({:ok, type, %{warnings: []}}), + def __typediag__!({:ok, type, %{warnings: []}}), do: raise("type checking ok without warnings: #{Descr.to_quoted_string(type)}") - def __typewarn__!({:ok, _type, %{warnings: warnings}}), + def __typediag__!({:ok, _type, %{warnings: warnings}}), do: raise("type checking ok but many warnings: #{inspect(warnings)}") - def __typewarn__!({:error, %{warnings: warnings}}), + def __typediag__!({:error, %{warnings: warnings}}), do: raise("type checking errored with warnings: #{inspect(warnings)}") + @doc false + def __typewarn__!(result) do + {type, %{message: message}} = __typediag__!(result) + {type, message} + end + @doc """ Building block for typechecking a given AST. """ diff --git a/lib/ex_unit/lib/ex_unit.ex b/lib/ex_unit/lib/ex_unit.ex index 8ba9379b96a..ca61ce78fbf 100644 --- a/lib/ex_unit/lib/ex_unit.ex +++ b/lib/ex_unit/lib/ex_unit.ex @@ -500,7 +500,7 @@ defmodule ExUnit do defp maybe_repeated_run(options, seed, load_us, repeat) do case ExUnit.Runner.run(options, load_us) do - {%{failures: 0}, {sync_modules, async_modules}} + {%{failures: 0}, {async_modules, sync_modules}} when repeat > 0 and (sync_modules != [] or async_modules != []) -> ExUnit.Server.restore_modules(async_modules, sync_modules) diff --git a/lib/ex_unit/lib/ex_unit/callbacks.ex b/lib/ex_unit/lib/ex_unit/callbacks.ex index 2669bb4d9a1..5705851bf32 100644 --- a/lib/ex_unit/lib/ex_unit/callbacks.ex +++ b/lib/ex_unit/lib/ex_unit/callbacks.ex @@ -535,15 +535,13 @@ defmodule ExUnit.Callbacks do test, as simply shutting down the process would cause it to be restarted according to its `:restart` value. - Another advantage is that the test process will act as both an ancestor - as well as a caller to the supervised processes. When a process is started - under a supervision tree, it typically populates the `$ancestors` key in - its process dictionary with all of its ancestors, which will include the test - process. Additionally, `start_supervised/2` will also store the test process - in the `$callers` key of the started process, allowing tools that perform - either ancestor or caller tracking to reach the test process. You can learn - more about these keys in - [the `Task` module](`Task#module-ancestor-and-caller-tracking`). + Finally, since Elixir v1.17.0, the test supervisor has both `$ancestors` + and `$callers` key in its process dictionary pointing to the test process. + This means developers can invoke `Process.get(:"$callers", [])` in their + `start_link` function and forward it to the spawned process, which may set + `Process.put(:"$callers", callers)` during its initialization. This may be + useful in projects who track process ownership during tests. You can learn + more about these keys in [the `Task` module](`Task#module-ancestor-and-caller-tracking`). """ @doc since: "1.5.0" @spec start_supervised(Supervisor.child_spec() | module | {module, term}, keyword) :: diff --git a/lib/ex_unit/lib/ex_unit/case.ex b/lib/ex_unit/lib/ex_unit/case.ex index 520a8e9e757..0537d4a49e4 100644 --- a/lib/ex_unit/lib/ex_unit/case.ex +++ b/lib/ex_unit/lib/ex_unit/case.ex @@ -578,7 +578,7 @@ defmodule ExUnit.Case do display. You can use `ExUnit.plural_rule/2` to set a custom pluralization. """ - @doc since: "1.10.0" + @doc since: "1.11.0" def register_test(mod, file, line, test_type, name, tags) do unless Module.has_attribute?(mod, :ex_unit_tests) do raise "cannot define #{test_type}. Please make sure you have invoked " <> diff --git a/lib/ex_unit/test/ex_unit/assertions_test.exs b/lib/ex_unit/test/ex_unit/assertions_test.exs index 14ccde889a9..2ab86672711 100644 --- a/lib/ex_unit/test/ex_unit/assertions_test.exs +++ b/lib/ex_unit/test/ex_unit/assertions_test.exs @@ -256,6 +256,28 @@ defmodule ExUnit.AssertionsTest do end end + test "assert match with __ENV__ in the pattern" do + message = + ExUnit.CaptureIO.capture_io(:stderr, fn -> + assert_raise CompileError, fn -> + Code.eval_string(""" + defmodule EnvMatch do + import ExUnit.Assertions + + def run do + assert __ENV__ = %{} + end + end + """) + end + end) + + assert message =~ "invalid pattern in match, __ENV__ is not allowed in matches" + after + :code.purge(EnvMatch) + :code.delete(EnvMatch) + end + test "assert match?" do true = assert match?({2, 1}, Value.tuple()) diff --git a/lib/iex/lib/iex/helpers.ex b/lib/iex/lib/iex/helpers.ex index 0662f83f3aa..1979fb83691 100644 --- a/lib/iex/lib/iex/helpers.ex +++ b/lib/iex/lib/iex/helpers.ex @@ -150,7 +150,7 @@ defmodule IEx.Helpers do reenable_tasks(config) force? = Keyword.get(options, :force, false) - args = ["--purge-consolidation-path-if-stale", "--return-errors", consolidation] + args = ["--purge-consolidation-path-if-stale", consolidation, "--return-errors"] args = if force?, do: ["--force" | args], else: args {result, _} = Mix.Task.run("compile", args) diff --git a/lib/logger/lib/logger/translator.ex b/lib/logger/lib/logger/translator.ex index f529616ed85..a2850047741 100644 --- a/lib/logger/lib/logger/translator.ex +++ b/lib/logger/lib/logger/translator.ex @@ -298,7 +298,7 @@ defmodule Logger.Translator do client_info: client, name: name, reason: {kind, reason, stack}, - state: {state, data}, + state: state, queue: queue, postponed: postponed, callback_mode: callback_mode, @@ -328,8 +328,6 @@ defmodule Logger.Translator do msg, "\nState: ", inspect(state, inspect_opts), - "\nData: ", - inspect(data, inspect_opts), "\nCallback mode: ", "#{inspect(callback_mode, inspect_opts)}, state_enter: #{state_enter?}" | format_client_info(client) diff --git a/lib/logger/test/logger/translator_test.exs b/lib/logger/test/logger/translator_test.exs index 331a4b8e125..e15010a291a 100644 --- a/lib/logger/test/logger/translator_test.exs +++ b/lib/logger/test/logger/translator_test.exs @@ -100,6 +100,28 @@ defmodule Logger.TranslatorTest do end end + defmodule MyGenStatemHandleEvent do + @behaviour :gen_statem + + @impl true + def callback_mode, do: :handle_event_function + + @impl true + def init(state) do + {:ok, :no_state, state} + end + + @impl true + def handle_event({:call, _}, :error, :no_state, _data) do + raise "oops" + end + + @impl :gen_statem + def format_status(_opts, [_pdict, _, state]) do + state + end + end + defmodule MyBridge do @behaviour :supervisor_bridge @@ -393,7 +415,7 @@ defmodule Logger.TranslatorTest do assert {%RuntimeError{message: "oops"}, [_ | _]} = process_metadata[:crash_reason] refute Map.has_key?(gen_statem_metadata, :initial_call) - assert process_metadata[:initial_call] == {Logger.TranslatorTest.MyGenStatem, :init, 1} + assert process_metadata[:initial_call] == {MyGenStatem, :init, 1} refute Map.has_key?(gen_statem_metadata, :registered_name) refute Map.has_key?(process_metadata, :registered_name) @@ -445,7 +467,7 @@ defmodule Logger.TranslatorTest do assert capture_log(:debug, fn -> catch_exit(:gen_statem.call(pid, :error)) end) =~ """ - [:ok, :ok, :ok, ...] + State: {:started, [:ok, ...]} """ after Application.put_env(:logger, :translator_inspect_opts, []) @@ -462,8 +484,7 @@ defmodule Logger.TranslatorTest do .* Queue: .* Postponed: \[\] - State: :started - Data: :ok + State: {:started, :ok} Callback mode: :state_functions, state_enter: false Client #PID<\d+\.\d+\.\d+> is alive .* @@ -488,8 +509,7 @@ defmodule Logger.TranslatorTest do .* Queue: .* Postponed: \[\] - State: :started - Data: :ok + State: {:started, :ok} Callback mode: :state_functions, state_enter: false Client :named_client is alive .* @@ -513,8 +533,7 @@ defmodule Logger.TranslatorTest do .* Queue: .* Postponed: \[\] - State: :started - Data: :ok + State: {:started, :ok} Callback mode: :state_functions, state_enter: false Client #PID<\d+\.\d+\.\d+> is dead """s @@ -533,8 +552,7 @@ defmodule Logger.TranslatorTest do .* Queue: .* Postponed: \[\] - State: :started - Data: :ok + State: {:started, :ok} Callback mode: :state_functions, state_enter: false """s end @@ -552,6 +570,34 @@ defmodule Logger.TranslatorTest do assert_receive {:event, {:string, ["Process " | _]}, _process_metadata} end + test "translates :gen_statem crashes when callback_mode is :handle_event_function" do + {:ok, pid} = :gen_statem.start(MyGenStatemHandleEvent, :ok, []) + + assert capture_log(:debug, fn -> + catch_exit(:gen_statem.call(pid, :error)) + end) =~ ~r""" + \[error\] :gen_statem #PID<\d+\.\d+\.\d+> terminating + \*\* \(RuntimeError\) oops + .* + Queue: .* + Postponed: \[\] + State: :ok + Callback mode: .*, state_enter: false + """s + + assert_receive {:event, {:string, [[":gen_statem " <> _ | _] | _]}, gen_statem_metadata} + assert_receive {:event, {:string, ["Process " | _]}, process_metadata} + + assert {%RuntimeError{message: "oops"}, [_ | _]} = gen_statem_metadata[:crash_reason] + assert {%RuntimeError{message: "oops"}, [_ | _]} = process_metadata[:crash_reason] + + refute Map.has_key?(gen_statem_metadata, :initial_call) + assert process_metadata[:initial_call] == {MyGenStatemHandleEvent, :init, 1} + + refute Map.has_key?(gen_statem_metadata, :registered_name) + refute Map.has_key?(process_metadata, :registered_name) + end + test "translates Task crashes" do {:ok, pid} = Task.start_link(__MODULE__, :task, [self()]) parent = self() @@ -902,13 +948,13 @@ defmodule Logger.TranslatorTest do test "translates Supervisor progress" do {:ok, pid} = Supervisor.start_link([], strategy: :one_for_one) - assert capture_log(:info, fn -> + assert capture_log(:debug, fn -> ref = Process.monitor(pid) Supervisor.start_child(pid, worker(Task, [__MODULE__, :sleep, [self()]])) Process.exit(pid, :normal) receive do: ({:DOWN, ^ref, _, _, _} -> :ok) end) =~ ~r""" - \[info\] Child Task of Supervisor #PID<\d+\.\d+\.\d+> \(Supervisor\.Default\) started + \[(debug|info)\] Child Task of Supervisor #PID<\d+\.\d+\.\d+> \(Supervisor\.Default\) started Pid: #PID<\d+\.\d+\.\d+> Start Call: Task.start_link\(Logger.TranslatorTest, :sleep, \[#PID<\d+\.\d+\.\d+>\]\) """ @@ -917,36 +963,36 @@ defmodule Logger.TranslatorTest do test "translates Supervisor progress with name" do {:ok, pid} = Supervisor.start_link([], strategy: :one_for_one, name: __MODULE__) - assert capture_log(:info, fn -> + assert capture_log(:debug, fn -> ref = Process.monitor(pid) Supervisor.start_child(pid, worker(Task, [__MODULE__, :sleep, [self()]])) Process.exit(pid, :normal) receive do: ({:DOWN, ^ref, _, _, _} -> :ok) end) =~ ~r""" - \[info\] Child Task of Supervisor Logger.TranslatorTest started + \[(debug|info)\] Child Task of Supervisor Logger.TranslatorTest started """ {:ok, pid} = Supervisor.start_link([], strategy: :one_for_one, name: {:global, __MODULE__}) - assert capture_log(:info, fn -> + assert capture_log(:debug, fn -> ref = Process.monitor(pid) Supervisor.start_child(pid, worker(Task, [__MODULE__, :sleep, [self()]])) Process.exit(pid, :normal) receive do: ({:DOWN, ^ref, _, _, _} -> :ok) end) =~ ~r""" - \[info\] Child Task of Supervisor Logger.TranslatorTest started + \[(debug|info)\] Child Task of Supervisor Logger.TranslatorTest started """ {:ok, pid} = Supervisor.start_link([], strategy: :one_for_one, name: {:via, :global, __MODULE__}) - assert capture_log(:info, fn -> + assert capture_log(:debug, fn -> ref = Process.monitor(pid) Supervisor.start_child(pid, worker(Task, [__MODULE__, :sleep, [self()]])) Process.exit(pid, :normal) receive do: ({:DOWN, ^ref, _, _, _} -> :ok) end) =~ ~r""" - \[info\] Child Task of Supervisor Logger.TranslatorTest started + \[(debug|info)\] Child Task of Supervisor Logger.TranslatorTest started """ end diff --git a/lib/mix/lib/mix/compilers/elixir.ex b/lib/mix/lib/mix/compilers/elixir.ex index a39a1abd024..1ccf936ec04 100644 --- a/lib/mix/lib/mix/compilers/elixir.ex +++ b/lib/mix/lib/mix/compilers/elixir.ex @@ -609,8 +609,8 @@ defmodule Mix.Compilers.Elixir do # within the dependency, they will be recompiled. However, export # and runtime dependencies won't have recompiled so we need to # propagate them to the parent app. - {dep_modules, _, _} = - fixpoint_runtime_modules(manifest_sources, Map.from_keys(dep_modules, true)) + dep_modules = + fixpoint_non_compile_modules(manifest_sources, Map.from_keys(dep_modules, true)) old_exports = Map.get(deps_exports, app, %{}) @@ -654,21 +654,19 @@ defmodule Mix.Compilers.Elixir do end end - defp fixpoint_runtime_modules(sources, modules) when modules != %{} do - fixpoint_runtime_modules(Map.to_list(sources), modules, false, [], [], sources) + defp fixpoint_non_compile_modules(sources, modules) when modules != %{} do + fixpoint_non_compile_modules(Map.to_list(sources), modules, false, []) end - defp fixpoint_runtime_modules(sources, modules) do - {modules, [], sources} + defp fixpoint_non_compile_modules(_sources, modules) do + modules end - defp fixpoint_runtime_modules( - [{source_path, source_entry} = pair | sources], + defp fixpoint_non_compile_modules( + [{_source_path, source_entry} = pair | sources], modules, new?, - pending_sources, - acc_modules, - acc_sources + pending_sources ) do source(export_references: export_refs, runtime_references: runtime_refs) = source_entry @@ -676,24 +674,19 @@ defmodule Mix.Compilers.Elixir do new_modules = Enum.reject(source(source_entry, :modules), &Map.has_key?(modules, &1)) modules = Enum.reduce(new_modules, modules, &Map.put(&2, &1, true)) new? = new? or new_modules != [] - acc_modules = new_modules ++ acc_modules - - acc_sources = - Map.replace!(acc_sources, source_path, source(source_entry, runtime_warnings: [])) - - fixpoint_runtime_modules(sources, modules, new?, pending_sources, acc_modules, acc_sources) + fixpoint_non_compile_modules(sources, modules, new?, pending_sources) else pending_sources = [pair | pending_sources] - fixpoint_runtime_modules(sources, modules, new?, pending_sources, acc_modules, acc_sources) + fixpoint_non_compile_modules(sources, modules, new?, pending_sources) end end - defp fixpoint_runtime_modules([], modules, new?, pending_sources, acc_modules, acc_sources) + defp fixpoint_non_compile_modules([], modules, new?, pending_sources) when new? == false or pending_sources == [], - do: {modules, acc_modules, acc_sources} + do: modules - defp fixpoint_runtime_modules([], modules, true, pending_sources, acc_modules, acc_sources), - do: fixpoint_runtime_modules(pending_sources, modules, false, [], acc_modules, acc_sources) + defp fixpoint_non_compile_modules([], modules, true, pending_sources), + do: fixpoint_non_compile_modules(pending_sources, modules, false, []) defp exports_md5(module, use_attributes?) do cond do @@ -1068,7 +1061,7 @@ defmodule Mix.Compilers.Elixir do end end - defp each_cycle(runtime_modules, compile_path, timestamp, state) do + defp each_cycle(stale_modules, compile_path, timestamp, state) do {modules, _exports, sources, changed, pending_modules, stale_exports} = state {pending_modules, exports, changed} = @@ -1081,11 +1074,33 @@ defmodule Mix.Compilers.Elixir do end if changed == [] do - # We merge runtime_modules (which is a map of %{module => true}) into - # a map of modules (which is a map of %{module => record}). This is fine - # since fixpoint_runtime_modules only cares about map keys. - {_, runtime_modules, sources} = - fixpoint_runtime_modules(sources, Map.merge(modules, runtime_modules)) + # We merge stale_modules (which is a map of %{module => true} that the user changed) + # into a map of modules we compiled (which is a map of %{module => record}). This is + # fine because we only care about the keys. + changed_modules = Map.merge(modules, stale_modules) + + # Now we do a simple pass finding anything that directly depends on the modules that + # changed. We don't need to compute a fixpoint, because now only the directly affected + # matter. + {sources, runtime_modules} = + Enum.reduce(sources, {sources, []}, fn + {source_path, source_entry}, {acc_sources, acc_modules} -> + source(export_references: export_refs, runtime_references: runtime_refs) = + source_entry + + if has_any_key?(changed_modules, export_refs) or + has_any_key?(changed_modules, runtime_refs) do + acc_sources = + Map.replace!(acc_sources, source_path, source(source_entry, runtime_warnings: [])) + + new_modules = + Enum.reject(source(source_entry, :modules), &Map.has_key?(changed_modules, &1)) + + {acc_sources, new_modules ++ acc_modules} + else + {acc_sources, acc_modules} + end + end) runtime_paths = Enum.map(runtime_modules, &{&1, Path.join(compile_path, Atom.to_string(&1) <> ".beam")}) diff --git a/lib/mix/lib/mix/generator.ex b/lib/mix/lib/mix/generator.ex index e2ccceb65ad..2551b8ecb38 100644 --- a/lib/mix/lib/mix/generator.ex +++ b/lib/mix/lib/mix/generator.ex @@ -27,6 +27,13 @@ defmodule Mix.Generator do if opts[:force] || overwrite?(path, contents) do File.mkdir_p!(Path.dirname(path)) + + contents = + case opts[:format_elixir] do + true -> [Code.format_string!(contents), ?\n] + _ -> contents + end + File.write!(path, contents) true else @@ -94,6 +101,7 @@ defmodule Mix.Generator do * `:force` - forces copying without a shell prompt * `:quiet` - does not log command output + * `:format_elixir` (since v1.18.0) - if `true`, apply formatter to the generated file ## Examples diff --git a/lib/mix/lib/mix/scm/git.ex b/lib/mix/lib/mix/scm/git.ex index ea8f1d7eebe..edc96c7ad3b 100644 --- a/lib/mix/lib/mix/scm/git.ex +++ b/lib/mix/lib/mix/scm/git.ex @@ -9,7 +9,7 @@ defmodule Mix.SCM.Git do @impl true def format(opts) do - if rev = get_opts_rev(opts) do + if rev = opts[:ref] || opts[:branch] || opts[:tag] do "#{redact_uri(opts[:git])} - #{rev}" else redact_uri(opts[:git]) @@ -22,7 +22,7 @@ defmodule Mix.SCM.Git do {:git, _, lock_rev, lock_opts} -> lock = String.slice(lock_rev, 0, 7) - case Enum.find_value([:branch, :ref, :tag], &List.keyfind(lock_opts, &1, 0)) do + case Enum.find_value([:ref, :branch, :tag], &List.keyfind(lock_opts, &1, 0)) do {:ref, _} -> lock <> " (ref)" {key, val} -> lock <> " (#{key}: #{val})" nil -> lock @@ -125,18 +125,18 @@ defmodule Mix.SCM.Git do sparse_toggle(opts) update_origin(opts[:git]) - rev = get_lock_rev(opts[:lock], opts) || get_opts_rev(opts) - # Fetch external data + branch_or_tag = opts[:branch] || opts[:tag] + ["--git-dir=.git", "fetch", "--force", "--quiet"] |> Kernel.++(progress_switch(git_version())) |> Kernel.++(tags_switch(opts[:tag])) |> Kernel.++(depth_switch(opts[:depth])) - |> Kernel.++(if rev, do: ["origin", rev], else: []) + |> Kernel.++(if branch_or_tag, do: ["origin", branch_or_tag], else: []) |> git!() # Migrate the Git repo - rev = rev || default_branch() + rev = get_lock_rev(opts[:lock], opts) || opts[:ref] || branch_or_tag || default_branch() git!(["--git-dir=.git", "checkout", "--quiet", rev]) if opts[:submodules] do @@ -250,18 +250,24 @@ defmodule Mix.SCM.Git do end defp validate_depth(opts) do - case Keyword.take(opts, [:depth]) do - [] -> - opts + case Keyword.take(opts, [:depth, :ref]) do + [_, _] -> + Mix.raise( + "Cannot specify :depth and :ref at the same time. " <> + "Error on Git dependency: #{redact_uri(opts[:git])}" + ) - [{:depth, depth}] when is_integer(depth) and depth > 0 -> + [depth: depth] when is_integer(depth) and depth > 0 -> opts - invalid_depth -> + [depth: invalid_depth] -> Mix.raise( "The depth must be a positive integer, and be specified only once, got: #{inspect(invalid_depth)}. " <> "Error on Git dependency: #{redact_uri(opts[:git])}" ) + + _ -> + opts end end @@ -290,10 +296,6 @@ defmodule Mix.SCM.Git do end end - defp get_opts_rev(opts) do - opts[:branch] || opts[:ref] || opts[:tag] - end - defp redact_uri(git) do case URI.parse(git) do %{userinfo: nil} -> git diff --git a/lib/mix/lib/mix/tasks/compile.leex.ex b/lib/mix/lib/mix/tasks/compile.leex.ex index 3594e060980..c4f7e47e090 100644 --- a/lib/mix/lib/mix/tasks/compile.leex.ex +++ b/lib/mix/lib/mix/tasks/compile.leex.ex @@ -66,8 +66,8 @@ defmodule Mix.Tasks.Compile.Leex do defp preload(project) do # TODO: Remove me in Elixir v2.0 unless :leex in List.wrap(project[:compilers]) do - IO.warn( - "in order to compile .xrl files, you must add \"compilers: [:leex] ++ Mix.compilers()\" to the \"def project\" section of #{project[:app]}'s mix.exs" + Mix.shell().error( + "warning: in order to compile .xrl files, you must add \"compilers: [:leex] ++ Mix.compilers()\" to the \"def project\" section of #{project[:app]}'s mix.exs" ) end diff --git a/lib/mix/lib/mix/tasks/compile.yecc.ex b/lib/mix/lib/mix/tasks/compile.yecc.ex index 22ad67f1df5..044f244e32d 100644 --- a/lib/mix/lib/mix/tasks/compile.yecc.ex +++ b/lib/mix/lib/mix/tasks/compile.yecc.ex @@ -67,8 +67,8 @@ defmodule Mix.Tasks.Compile.Yecc do defp preload(project) do # TODO: Remove me in Elixir v2.0 unless :yecc in List.wrap(project[:compilers]) do - IO.warn( - "in order to compile .yrl files, you must add \"compilers: [:yecc] ++ Mix.compilers()\" to the \"def project\" section of #{project[:app]}'s mix.exs" + Mix.shell().error( + "warning: in order to compile .yrl files, you must add \"compilers: [:yecc] ++ Mix.compilers()\" to the \"def project\" section of #{project[:app]}'s mix.exs" ) end diff --git a/lib/mix/lib/mix/tasks/deps.ex b/lib/mix/lib/mix/tasks/deps.ex index d972123fdec..c3a5f7faad2 100644 --- a/lib/mix/lib/mix/tasks/deps.ex +++ b/lib/mix/lib/mix/tasks/deps.ex @@ -125,7 +125,8 @@ defmodule Mix.Tasks.Deps do * `:depth` *(since v1.17.0)* - creates a shallow clone of the Git repository, limiting the history to the specified number of commits. This can significantly improve clone speed for large repositories when full history is not needed. - The value must be a positive integer, typically `1`. + The value must be a positive integer, typically `1`. Cannot be used with the + `:ref` option. If your Git repository requires authentication, such as basic username:password HTTP authentication via URLs, it can be achieved via Git configuration, keeping diff --git a/lib/mix/lib/mix/tasks/profile.eprof.ex b/lib/mix/lib/mix/tasks/profile.eprof.ex index 14ff7413f7f..fb25222fe17 100644 --- a/lib/mix/lib/mix/tasks/profile.eprof.ex +++ b/lib/mix/lib/mix/tasks/profile.eprof.ex @@ -224,9 +224,9 @@ defmodule Mix.Tasks.Profile.Eprof do calls_opt = Keyword.get(opts, :calls, 0) time_opt = Keyword.get(opts, :time, 0) - call_results - |> Stream.filter(fn {_mfa, {count, _time}} -> count >= calls_opt end) - |> Stream.filter(fn {_mfa, {_count, time}} -> time >= time_opt end) + Enum.filter(call_results, fn {_mfa, {count, time}} -> + count >= calls_opt and time >= time_opt + end) end defp sort_results(call_results, opts) do @@ -307,15 +307,10 @@ defmodule Mix.Tasks.Profile.Eprof do max_lengths = Enum.map(header, &String.length/1) Enum.reduce(rows, max_lengths, fn row, max_lengths -> - Stream.map(row, &String.length/1) - |> Stream.zip(max_lengths) - |> Enum.map(&max/1) + Enum.zip_with(row, max_lengths, fn cell, length -> String.length(cell) |> max(length) end) end) end - defp max({a, b}) when a >= b, do: a - defp max({_, b}), do: b - @format "~-*s ~*s ~*s ~*s ~*s~n" defp print_row(row, column_lengths) do diff --git a/lib/mix/lib/mix/tasks/profile.tprof.ex b/lib/mix/lib/mix/tasks/profile.tprof.ex new file mode 100644 index 00000000000..fde9c5a3859 --- /dev/null +++ b/lib/mix/lib/mix/tasks/profile.tprof.ex @@ -0,0 +1,455 @@ +defmodule Mix.Tasks.Profile.Tprof do + use Mix.Task + + @shortdoc "Profiles the given file or expression with tprof" + + @moduledoc """ + Profiles the given file or expression using Erlang's `tprof` tool. + + Requires Erlang/OTP27 or above. + + [`:tprof`](`:tprof`) is an experimental module introduced in Erlang/OTP 27 which + provides a unified API for measuring call count, time, and allocation, and aims to + replace [`:eprof`](`:eprof`) and [`:cprof`](`:cprof`). + It can be useful when you want to discover the bottlenecks related to any of these + measurements. + + Before running the code, it invokes the `app.start` task which compiles + and loads your project. After that, the target expression is profiled together + with all matching function calls using the Erlang trace BIFs. The tracing of + the function calls for that is enabled when the profiling is begun, and + disabled when profiling is stopped. + + To profile the code, you can use syntax similar to the `mix run` task: + + $ mix profile.tprof -e Hello.world + $ mix profile.tprof -e "[1, 2, 3] |> Enum.reverse |> Enum.map(&Integer.to_string/1)" + $ mix profile.tprof my_script.exs arg1 arg2 arg3 + + By default, tprof uses the `time` type, but you can profile memory too: + + $ mix profile.tprof -e "Enum.map([1, 2, 3], &Integer.to_string/1)" --type memory + + Call count is present with both type `time` and `memory`, but if you only need + the call count information, you can use the type `calls` which has the lowest footprint: + + $ mix profile.tprof -e "Enum.map([1, 2, 3], &Integer.to_string/1)" --type calls + + This task is automatically re-enabled, so you can profile multiple times + in the same Mix invocation. + + ## Command line options + + * `--matching` - only profile calls matching the given `Module.function/arity` pattern + * `--type` - the type of profiling, `calls`, `time` or `memory` (default: `time`) + * `--calls` - filters out any results with a call count lower than this + * `--time` - filters out any results that took lower than specified (in µs), the `type` needs to be `time` + * `--memory` - filters out any results that used less memory than specified (in words), the `type` needs to be `memory` + * `--sort` - sorts the results by `calls`, `per_call` or by the value of `type` (default: the value of `type`) + * `--report` - returns the per-process breakdown when `process`, or the total for all processes when `total` (default: `process`). + Always `total` when `type` is `calls`. + * `--eval`, `-e` - evaluates the given code + * `--require`, `-r` - requires pattern before running the command + * `--parallel`, `-p` - makes all requires parallel + * `--no-warmup` - skips the warmup step before profiling + * `--no-compile` - does not compile even if files require compilation + * `--no-deps-check` - does not check dependencies + * `--no-archives-check` - does not check archives + * `--no-halt` - does not halt the system after running the command + * `--no-start` - does not start applications after compilation + * `--no-elixir-version-check` - does not check the Elixir version from mix.exs + + ## Profile output + + Example output (`time` type): + + Profile results of #PID<0.107.0> + # CALLS % TIME µS/CALL + Total 20 100.00 2 0.10 + String.Chars.Integer.to_string/1 5 0.00 0 0.00 + anonymous fn/0 in :elixir_compiler_1.__FILE__/1 1 0.00 0 0.00 + Enum.each/2 1 0.00 0 0.00 + Enum.reduce_range/5 3 0.00 0 0.00 + :erlang.integer_to_binary/1 5 50.00 1 0.20 + anonymous fn/3 in Enum.each/2 5 50.00 1 0.20 + + Profile done over 6 matching functions + + Example output (`memory` type): + + Profile results of #PID<0.107.0> + # CALLS % WORDS PER CALL + Total 6 100.00 19 3.17 + Enum.each/2 1 21.05 4 4.00 + :erlang.integer_to_binary/1 5 78.95 15 3.00 + + Profile done over 2 matching functions + + Example output (`calls` type) + + Profile results over all processes + # CALLS % + Total 20 100.00 + anonymous fn/0 in :elixir_compiler_1.__FILE__/1 1 5.00 + Enum.each/2 1 5.00 + Enum.reduce_range/5 3 15.00 + :erlang.integer_to_binary/1 5 25.00 + String.Chars.Integer.to_string/1 5 25.00 + anonymous fn/3 in Enum.each/2 5 25.00 + + Profile done over 6 matching functions + + The default output contains data gathered from all matching functions. The first + row after the header contains the sums of the partial results and the average time + or memory usage for all the function calls listed. + The following rows contain the function call, followed by the number of times that + the function was called, then by the percentage of time/memory that the call uses, + then the total time/memory for that function in microseconds/words, and, finally, + the average time/memory per call in microseconds/words. + + When `--matching` option is specified, call count tracing will be started only for + the functions matching the given pattern: + + Profile results of #PID<0.106.0> + # CALLS % TIME µS/CALL + Total 5 100.00 1 0.20 + String.Chars.Integer.to_string/1 5 100.00 1 0.20 + + Profile done over 1 matching functions + + The pattern can be a module name, such as `String` to count all calls to that module, + a call without arity, such as `String.split`, to count all calls to that function + regardless of arity, or a call with arity, such as `String.split/3`, to count all + calls to that exact module, function and arity. + + ## Caveats + + You should be aware that the code being profiled is running in an anonymous + function which is invoked by [`:tprof` module](https://www.erlang.org/doc/man/tprof.html). + Thus, you'll see some additional entries in your profile output. It is also + important to note that the profiler is stopped as soon as the code has finished running, + and this may need special attention, when: running asynchronous code as function calls which were + called before the profiler stopped will not be counted; running synchronous code as long + running computations and a profiler without a proper MFA trace pattern or filter may + lead to a result set which is difficult to comprehend. + + You should expect a slowdown in your code execution using this tool since `:tprof` has + some performance impact on the execution, but the impact is considerably lower than + `Mix.Tasks.Profile.Fprof`. If you have a large system try to profile a limited + scenario or focus on the main modules or processes. The `calls` type can also be used, + which is more limited but has a lower footprint. + """ + + @switches [ + parallel: :boolean, + require: :keep, + eval: :keep, + config: :keep, + matching: :string, + halt: :boolean, + compile: :boolean, + deps_check: :boolean, + type: :string, + calls: :integer, + time: :integer, + memory: :integer, + sort: :string, + report: :string, + start: :boolean, + archives_check: :boolean, + warmup: :boolean, + elixir_version_check: :boolean, + parallel_require: :keep + ] + + @aliases [ + r: :require, + p: :parallel, + e: :eval, + c: :config + ] + + @impl true + def run(args) do + {opts, head} = OptionParser.parse_head!(args, aliases: @aliases, strict: @switches) + Mix.Task.reenable("profile.tprof") + + Mix.Tasks.Run.run( + ["--no-mix-exs" | args], + opts, + head, + &profile_code(&1, opts), + &profile_code(File.read!(&1), opts) + ) + end + + defp profile_code(code_string, opts) do + opts = Enum.map(opts, &parse_opt/1) + + content = + quote do + unquote(__MODULE__).profile( + fn -> + unquote(Code.string_to_quoted!(code_string)) + end, + unquote(Macro.escape(opts)) + ) + end + + # Use compile_quoted since it leaves less noise than eval_quoted + Code.compile_quoted(content) + end + + defp parse_opt({:matching, matching}) do + case Mix.Utils.parse_mfa(matching) do + {:ok, [m, f, a]} -> {:matching, {m, f, a}} + {:ok, [m, f]} -> {:matching, {m, f, :_}} + {:ok, [m]} -> {:matching, {m, :_, :_}} + :error -> Mix.raise("Invalid matching pattern: #{matching}") + end + end + + defp parse_opt({:type, "time"}), do: {:type, :time} + defp parse_opt({:type, "calls"}), do: {:type, :calls} + defp parse_opt({:type, "memory"}), do: {:type, :memory} + defp parse_opt({:type, other}), do: Mix.raise("Invalid type option: #{other}") + + defp parse_opt({:report, "process"}), do: {:report, :process} + defp parse_opt({:report, "total"}), do: {:report, :total} + defp parse_opt({:report, other}), do: Mix.raise("Invalid report option: #{other}") + + defp parse_opt({:sort, "time"}), do: {:sort, :time} + defp parse_opt({:sort, "calls"}), do: {:sort, :calls} + defp parse_opt({:sort, "memory"}), do: {:sort, :memory} + defp parse_opt({:sort, "per_call"}), do: {:sort, :per_call} + defp parse_opt({:sort, other}), do: Mix.raise("Invalid sort option: #{other}") + defp parse_opt(other), do: other + + @doc """ + Allows to programmatically run the `tprof` profiler on expression in `fun`. + + Returns the return value of `fun`. + + ## Options + + * `:matching` - only profile calls matching the given pattern in form of + `{module, function, arity}`, where each element may be replaced by `:_` + to allow any value + * `:type` - the type of profiling, possible values are `:time`, `:memory` or `:calls`, + (default: `:time`), see [moduledoc](`Mix.Tasks.Profile.Tprof`) for more information + + * `:calls` - filters out any results with a call count lower than this + * `:time` - filters out any results that took lower than specified (in µs), + `type` needs to be `:time` + * `:memory` - filters out any results that used less memory than specified (in words), + `type` needs to be `:memory` + * `:sort` - sort the results by `:calls`, `:per_call` or by the value of `type` + (default: the value of `type`) + * `:report` - returns the per-process breakdown when `:process`, or the total for all + processes when `:total` (default: `:process`). Always `:total` when `type` is `:calls`. + * `:warmup` - if the code should be warmed up before profiling (default: `true`) + * `:set_on_spawn` - if newly spawned processes should be measured (default: `true`) + + """ + @spec profile((-> result), keyword()) :: result when result: any() + def profile(fun, opts \\ []) when is_function(fun, 0) do + Mix.ensure_application!(:tools) + {type, return_value, results} = profile_and_analyse(fun, opts) + print_output(type, results) + return_value + end + + defp profile_and_analyse(fun, opts) do + if Keyword.get(opts, :warmup, true) do + IO.puts("Warmup...\n") + fun.() + end + + matching = Keyword.get(opts, :matching, {:_, :_, :_}) + set_on_spawn = Keyword.get(opts, :set_on_spawn, true) + type = Keyword.get(opts, :type, :time) + report = Keyword.get(opts, :report, :process) + + sort_by = + case Keyword.get(opts, :sort) do + nil -> + :measurement + + :calls -> + :calls + + :per_call -> + :measurement_per_call + + ^type -> + :measurement + + other -> + Mix.raise("Incompatible sort option #{inspect(other)} with type #{inspect(type)}") + end + + tprof_type = to_tprof_type(type) + + {return_value, {^tprof_type, traces}} = + tprof_module().profile(fun, %{ + set_on_spawn: set_on_spawn, + pattern: matching, + type: tprof_type, + report: :return + }) + + inspected = tprof_module().inspect({tprof_type, traces}, report, sort_by) + + results = + inspected + |> Enum.map(fn {pid, {^tprof_type, measurement_total, call_results}} -> + parsed_calls = + call_results + |> filter_results(type, opts) + |> add_totals(measurement_total) + + {pid, parsed_calls} + end) + + {type, return_value, results} + end + + defp to_tprof_type(:calls), do: :call_count + defp to_tprof_type(:time), do: :call_time + defp to_tprof_type(:memory), do: :call_memory + + defp filter_results(call_results, type, opts) do + calls_opt = Keyword.get(opts, :calls, 0) + + measurement_opt = + get_filter_value!(type, Keyword.get(opts, :time), Keyword.get(opts, :memory)) + + Enum.filter(call_results, fn {_module, _fa, count, measurement, _, _} -> + count >= calls_opt and measurement >= measurement_opt + end) + end + + defp get_filter_value!(type, time, _memory) when is_integer(time) and type != :time do + Mix.raise("Incompatible use of time option with type #{inspect(type)}") + end + + defp get_filter_value!(type, _time, memory) when is_integer(memory) and type != :memory do + Mix.raise("Incompatible use of memory option with type #{inspect(type)}") + end + + defp get_filter_value!(:time, time, nil) when is_integer(time), do: time + defp get_filter_value!(:memory, nil, memory) when is_integer(memory), do: memory + + defp get_filter_value!(_, nil, nil), do: 0 + + defp add_totals(call_results, measurement_total) do + {function_count, calls} = + Enum.reduce(call_results, {0, 0}, fn {_mod, _fa, count, _, _, _}, acc -> + {function_count, calls} = acc + {function_count + 1, calls + count} + end) + + {function_count, call_results, calls, measurement_total} + end + + defp print_output(_type, []) do + print_function_count(0) + end + + defp print_output(type, results) do + Enum.each(results, &print_result(type, &1)) + end + + defp print_result(type, {pid, {function_count, call_results, calls, total_measurement}}) do + header = header(type) + + formatted_rows = Enum.map(call_results, &format_row/1) + formatted_total = format_total(total_measurement, calls) + + column_lengths = column_lengths(header, [formatted_total | formatted_rows]) + + IO.puts("") + + print_pid_row(pid) + print_row(header, column_lengths, type) + print_row(formatted_total, column_lengths, type) + Enum.each(formatted_rows, &print_row(&1, column_lengths, type)) + + IO.puts("") + + print_function_count(function_count) + end + + defp header(:calls), do: ["#", "CALLS", "%"] + defp header(:time), do: ["#", "CALLS", "%", "TIME", "µS/CALL"] + defp header(:memory), do: ["#", "CALLS", "%", "WORDS", "PER CALL"] + + defp print_pid_row(:all) do + IO.puts("Profile results over all processes") + end + + defp print_pid_row(pid) when is_pid(pid) do + IO.puts("Profile results of #{inspect(pid)}") + end + + defp format_row({module, {function, arity}, count, measurement, per_call, percentage}) do + mfa = Exception.format_mfa(module, function, arity) + percentage = :erlang.float_to_binary(percentage, [{:decimals, 2}]) + per_call = :erlang.float_to_binary(per_call, [{:decimals, 2}]) + count = Integer.to_string(count) + measurement = Integer.to_string(measurement) + + [mfa, count, percentage, measurement, per_call] + end + + defp format_total(total_measurement, total_count) do + per_call = :erlang.float_to_binary(divide(total_measurement, total_count), [{:decimals, 2}]) + + [ + "Total", + Integer.to_string(total_count), + "100.00", + Integer.to_string(total_measurement), + per_call + ] + end + + defp divide(_, 0), do: 0.0 + defp divide(t, n), do: t / n + + defp column_lengths(header, rows) do + max_lengths = Enum.map(header, &String.length/1) + + Enum.reduce(rows, max_lengths, fn row, max_lengths -> + Enum.zip_with(row, max_lengths, fn cell, length -> String.length(cell) |> max(length) end) + end) + end + + @call_format "~-*s ~*s ~*s~n" + @measurement_format "~-*s ~*s ~*s ~*s ~*s~n" + + defp print_row(row, column_lengths, type) do + to_print = + column_lengths + |> Stream.zip(Stream.map(row, &String.to_charlist/1)) + |> Enum.flat_map(&Tuple.to_list/1) + + case type do + :calls -> :io.format(@call_format, to_print) + _ -> :io.format(@measurement_format, to_print) + end + end + + defp print_function_count(count) do + IO.puts("Profile done over #{count} matching functions") + end + + # TODO remove once we require Erlang/OTP 27+ + defp tprof_module do + if Code.ensure_loaded?(:tprof) do + :tprof + else + Mix.raise("mix profile.tprof requires Erlang/OTP 27 or above") + end + end +end diff --git a/lib/mix/lib/mix/tasks/release.ex b/lib/mix/lib/mix/tasks/release.ex index 181579b858c..9341cc2a030 100644 --- a/lib/mix/lib/mix/tasks/release.ex +++ b/lib/mix/lib/mix/tasks/release.ex @@ -402,8 +402,8 @@ defmodule Mix.Tasks.Release do If you are setting this option manually, we recommend the cookie option to be a long and randomly generated string, such as: - `Base.url_encode64(:crypto.strong_rand_bytes(40))`. We also recommend to restrict - the characters in the cookie to the subset returned by `Base.url_encode64/1`. + `Base.encode32(:crypto.strong_rand_bytes(40))`. We also recommend restricting + the characters in the cookie to only alphanumeric characters and underscore. * `:validate_compile_env` - by default a release will match all runtime configuration against any configuration that was marked at compile time @@ -601,7 +601,7 @@ defmodule Mix.Tasks.Release do -start_epmd false -erl_epmd_port 6789 # In remote.vm.args.eex - # -start_epmd false -erl_epmd_port 6789 -dist_listen false + -start_epmd false -erl_epmd_port 6789 -dist_listen false You can pick any port of your choice. @@ -833,9 +833,9 @@ defmodule Mix.Tasks.Release do * `RELEASE_DISTRIBUTION` - how do we want to run the distribution. May be `name` (long names), `sname` (short names) or `none` - (distribution is not started automatically). Defaults to - `sname` which allows access only within the current system. - `name` allows external connections + (distribution is not started automatically). Defaults to `sname`. + When connecting nodes across hosts, you typically want to set + this to `name` (required to use IPs as host names) * `RELEASE_BOOT_SCRIPT` - the name of the boot script to use when starting the release. This script is used when running commands such as `start` and diff --git a/lib/mix/lib/mix/tasks/release.init.ex b/lib/mix/lib/mix/tasks/release.init.ex index 285a0581c98..c983f3c3304 100644 --- a/lib/mix/lib/mix/tasks/release.init.ex +++ b/lib/mix/lib/mix/tasks/release.init.ex @@ -53,7 +53,7 @@ defmodule Mix.Tasks.Release.Init do ## Enable deployment without epmd ## (requires changing both vm.args and remote.vm.args) - ##-start_epmd false -erl_epmd_port 6789#{remote? && " -dist_listen false"} + ##-start_epmd false -erl_epmd_port 6789#{if(remote?, do: " -dist_listen false")} """ @doc false @@ -136,7 +136,7 @@ defmodule Mix.Tasks.Release.Init do ;; *) - echo "ERROR: Expected sname, name, or none in RELEASE_DISTRIBUTION, got: $RELEASE_DISTRIBUTION" >&2 + echo "ERROR: Expected RELEASE_DISTRIBUTION to be sname, name, or none, got: $RELEASE_DISTRIBUTION" >&2 exit 1 ;; esac @@ -280,7 +280,7 @@ defmodule Mix.Tasks.Release.Init do rem set RELEASE_MODE=interactive rem Set the release to work across nodes. - rem RELEASE_DISTRIBUTION must be "sname" (local), "name" (distributed) or "none". + rem RELEASE_DISTRIBUTION must be sname (local), name (distributed) or none. rem set RELEASE_DISTRIBUTION=name rem set RELEASE_NODE=<%= @release.name %> """ @@ -314,13 +314,33 @@ defmodule Mix.Tasks.Release.Init do if not defined RELEASE_BOOT_SCRIPT_CLEAN (set RELEASE_BOOT_SCRIPT_CLEAN=start_clean) if not defined RELEASE_SYS_CONFIG (set RELEASE_SYS_CONFIG=!REL_VSN_DIR!\sys) + if "!RELEASE_DISTRIBUTION!" == "none" ( + rem + ) else if "!RELEASE_DISTRIBUTION!" == "name" ( + rem + ) else if "!RELEASE_DISTRIBUTION!" == "sname" ( + rem + ) else ( + echo ERROR: Expected RELEASE_DISTRIBUTION to be sname, name, or none, got: !RELEASE_DISTRIBUTION! + exit /B 1 + ) + + if "!RELEASE_MODE!" == "embedded" ( + rem + ) else if "!RELEASE_MODE!" == "interactive" ( + rem + ) else ( + echo ERROR: Expected RELEASE_MODE to be embedded or interactive, got: !RELEASE_MODE! + exit /B 1 + ) + if "%~1" == "start" (set "REL_EXEC=elixir" && set "REL_EXTRA=--no-halt" && set "REL_GOTO=start") if "%~1" == "start_iex" (set "REL_EXEC=iex" && set "REL_EXTRA=--werl" && set "REL_GOTO=start") if "%~1" == "install" (set "REL_GOTO=install") if "%~1" == "eval" ( if "%~2" == "" ( echo ERROR: EVAL expects an expression as argument - goto end + exit /B 1 ) set "REL_GOTO=eval" ) @@ -348,7 +368,7 @@ defmodule Mix.Tasks.Release.Init do if "%~1" == "rpc" ( if "%~2" == "" ( echo ERROR: RPC expects an expression as argument - goto end + exit /B 1 ) set "REL_RPC=%~2" goto rpc @@ -369,7 +389,10 @@ defmodule Mix.Tasks.Release.Init do echo pid Prints the operating system PID of the running system via a remote command echo version Prints the release name and version to be booted echo. - if not "%~1" == "" (echo ERROR: Unknown command %~1) + if not "%~1" == "" ( + echo ERROR: Unknown command %~1 + exit /B 1 + ) goto end :start @@ -427,7 +450,7 @@ defmodule Mix.Tasks.Release.Init do if "!RELEASE_DISTRIBUTION!" == "none" ( set RELEASE_DISTRIBUTION_FLAG= ) else ( - set RELEASE_DISTRIBUTION_FLAG=--!RELEASE_DISTRIBUTION! "rem-!RANDOM!-!RELEASE_NODE!" + set RELEASE_DISTRIBUTION_FLAG=--!RELEASE_DISTRIBUTION! "rpc-!RANDOM!-!RELEASE_NODE!" ) "!REL_VSN_DIR!\elixir.bat" ^ @@ -452,7 +475,7 @@ defmodule Mix.Tasks.Release.Init do if "!RELEASE_DISTRIBUTION!" == "none" ( echo ERROR: RELEASE_DISTRIBUTION is required in install command - goto end + exit /B 1 ) "!ERLSRV!" add "!RELEASE_NAME!_!RELEASE_NAME!" ^ diff --git a/lib/mix/lib/mix/tasks/test.coverage.ex b/lib/mix/lib/mix/tasks/test.coverage.ex index 6e43c14fd27..4d86d6ddddb 100644 --- a/lib/mix/lib/mix/tasks/test.coverage.ex +++ b/lib/mix/lib/mix/tasks/test.coverage.ex @@ -288,9 +288,17 @@ defmodule Mix.Tasks.Test.Coverage do output = Keyword.get(opts, :output, "cover") File.mkdir_p!(output) - for mod <- modules do - {:ok, _} = :cover.analyse_to_file(mod, ~c"#{output}/#{mod}.html", [:html]) - end + modules + |> Enum.map(fn mod -> + pid = :cover.async_analyse_to_file(mod, ~c"#{output}/#{mod}.html", [:html]) + Process.monitor(pid) + end) + |> Enum.each(fn ref -> + receive do + {:DOWN, ^ref, :process, _pid, _reason} -> + :ok + end + end) Mix.shell().info("Generated HTML coverage results in #{inspect(output)} directory") end diff --git a/lib/mix/lib/mix/tasks/xref.ex b/lib/mix/lib/mix/tasks/xref.ex index fc12d4dd23d..d43b5819db2 100644 --- a/lib/mix/lib/mix/tasks/xref.ex +++ b/lib/mix/lib/mix/tasks/xref.ex @@ -14,16 +14,111 @@ defmodule Mix.Tasks.Xref do $ mix xref MODE - All available modes are discussed below. + All available modes are discussed below, after a brief + introduction to xref. This task is automatically re-enabled, so you can print information multiple times in the same Mix invocation. - ## mix xref callers MODULE + ## A brief introduction to xref - Prints all callers of the given module. Example: + The goal of `xref` is to analyze the dependencies between modules + and files. It is most commonly used to find problematic areas where + touching one file in a project causes a large subset of the project + to recompile. The most common cause of these problems are the so-called + "compile-connected" files. Those are files you depend on at compile-time + (for example, by invoking its macro or using it in the body of amodule) + which also have their own dependencies. - $ mix xref callers MyMod + Therefore, if your goal is to reduce recompilations, the first step is to run: + + $ mix xref graph --format stats --label compile-connected + + This command will show general information about the project, but + focus on compile-connected dependencies. In the stats, you will see + the following report: + + Top 10 files with most incoming dependencies: + * lib/livebook_web.ex (97) + * lib/livebook/config.ex (3) + * proto/lib/livebook_proto/deployment_group.pb.ex (2) + * lib/livebook_web/plugs/memory_provider.ex (2) + * proto/lib/livebook_proto/user_connected.pb.ex (1) + + You can see the first file, "lib/livebook_web.ex", is depended on by 97 + other files and, because we are using compile-connected, it also means + that "lib/livebook_web.ex" itself has its own dependencies. We can find + which files depend on "lib/livebook_web.ex" at compile time like this: + + $ mix xref graph --sink lib/livebook_web.ex --label compile --only-nodes + + And you can find the files lib/livebook_web.ex depends on like this: + + $ mix xref graph --source lib/livebook_web.ex --only-nodes + + The trouble here is precisely that, if any of the files in the latter + command changes, all of the files in the first command will be recompiled, + because compile time dependencies are transitive. + + Having compile time dependencies is a common feature in Elixir projects. + However, the modules you depend on at compile-time must avoid runtime + dependencies within the same project. You can understand all of the + dependencies of a given file by running: + + $ mix xref trace lib/livebook_web.ex + + The command above will output three types of dependencies, which we + detail next. + + ### Dependency types + + Elixir tracks three types of dependencies between modules: compile, + exports, and runtime. If a module has a compile time dependency on + another module, the caller module has to be recompiled whenever the + callee changes (or any runtime dependency of the callee changes). + Let's see an example: + + # lib/a.ex + defmodule A do + @hello B.hello() + def hello, do: @hello + end + + # lib/b.ex + defmodule B do + def hello, do: "hello" + def world, do: C.world() + end + + # lib/c.ex + defmodule C do + def world, do: "world" + end + + If `C.world/0` changes, `B` is marked as stale. `B` does not need to + be recompiled, because it depends on `C` at runtime, but anything that + depends on `B` at compile-time has to recompile, and that includes `A`. + + Compile-time dependencies are typically added when using macros or + when invoking functions in the module body (outside of functions). + This type of transitive compile-time dependencies, such as `A` + depending on `C` at compile-time through `B`, are called compile-connected. + + Export dependencies are compile time dependencies on the module API, + namely structs and its public definitions. For example, if you import + a module but only use its functions, it is an export dependency. If + you use a struct, it is an export dependency too. Export dependencies + are only recompiled if the module API changes. Note, however, that compile + time dependencies have higher precedence than exports. Therefore if + you import a module and use its macros, it is a compile time dependency. + + Runtime dependencies are added whenever you invoke another module + inside a function. Modules with runtime dependencies do not have + to be compiled when the callee changes, unless there is a transitive + compile or an outdated export time dependency between them. + + Over the next sections, we will explain what which `mix xref` command + does in detail. ## mix xref trace FILE @@ -87,14 +182,13 @@ defmodule Mix.Tasks.Xref do * `--exclude` - path to exclude. Can be repeated to exclude multiple paths. * `--label` - only shows relationships with the given label. - The labels are "compile", "export" and "runtime". By default, - the `--label` option simply filters the printed graph to show - only relationships with the given label. You can pass `--only-direct` - to trim the graph to only the nodes that have the direct - relationship given by label. There is also a special label - called "compile-connected" that keeps only compile-time files - with at least one transitive dependency. See "Dependency types" - section below. + The labels are "compile", "export" and "runtime". By default, the `--label` + option does not change how the graph is computed, it simply filters the + printed graph to show only relationships with the given label. However, + you can pass `--only-direct` to trim the graph to only the nodes that + have the direct relationship given by label. There is also a special + label called "compile-connected" that keeps only compile-time files with + at least one transitive dependency. See "Dependency types" section below. * `--group` - provide comma-separated paths to consider as a group. Dependencies from and into multiple files of the group are considered a single dependency. @@ -148,8 +242,8 @@ defmodule Mix.Tasks.Xref do those options with `--label` and `--only-nodes` to get all files that exhibit a certain property, for example: - # To show all compile-time relationships - $ mix xref graph --label compile + # To show all compile-connected relationships + $ mix xref graph --label compile-connected # To get the tree that depend on lib/foo.ex at compile time $ mix xref graph --label compile --sink lib/foo.ex @@ -163,6 +257,9 @@ defmodule Mix.Tasks.Xref do # To show general statistics about the graph $ mix xref graph --format stats + # To show all cycles with at least one compile-time dependency + $ mix xref graph --format cycles --label compile-connected + ### Understanding the printed graph When `mix xref graph` runs, it will print a tree of the following @@ -204,10 +301,9 @@ defmodule Mix.Tasks.Xref do The `--label compile` flag removes all non-compile dependencies. However, this can be misleading because having direct compile time dependencies is - not necessarily an issue. The biggest concern, as mentioned above, are the - transitive compile time dependencies. You can get all compile time - dependencies that cause transitive compile time dependencies by using - `--label compile-connected`: + not necessarily an issue. The biggest concern are the transitive compile + time dependencies. You can get all compile time dependencies that cause + transitive compile time dependencies by using `--label compile-connected`: $ mix xref graph --label compile-connected lib/a.ex @@ -235,56 +331,35 @@ defmodule Mix.Tasks.Xref do command will list all files from all umbrella children, without any namespacing. - ### Dependency types + ### Understanding the printed cycle - Elixir tracks three types of dependencies between modules: compile, - exports, and runtime. If a module has a compile time dependency on - another module, the caller module has to be recompiled whenever the - callee changes (or any runtime dependency of the callee changes). - Let's see an example: + If you run `mix xref graph --format cycle`, Elixir will print cycles + of shape: - # lib/a.ex - defmodule A do - @hello B.hello() - def hello, do: @hello - end + Cycle of length 3: - # lib/b.ex - defmodule B do - def hello, do: "hello" - def world, do: C.world() - end + lib/c.ex + lib/b.ex + lib/a.ex - # lib/c.ex - defmodule C do - def world, do: "world" - end + The cycles are given in order: `c.ex` depends on `b.ex` which depends + on `a.ex` which depends on `c.ex`. In particular, you want to avoid + cycles with compile dependencies in there. You can find those cycles + with: - If `C.world/0` changes, `B` is marked as stale. `B` does not need to - be recompiled, because it depends on `C` at runtime, but anything that - depends on `B` at compile-time has to recompile, and that includes `A`. + $ mix xref graph --format cycles --label compile-connected - Compile-time dependencies are typically added when using macros or - when invoking functions in the module body (outside of functions). - You can list all dependencies in a file by running - `mix xref trace path/to/file.ex`. This type of transitive compile-time - dependencies, such as `A` depending on `C` at compile-time through `B`, - can be found with the "compile-connected" label, as in - `mix xref graph --label compile-connected`. + Which may look like this: - Export dependencies are compile time dependencies on the module API, - namely structs and its public definitions. For example, if you import - a module but only use its functions, it is an export dependency. If - you use a struct, it is an export dependency too. Export dependencies - are only recompiled if the module API changes. Note, however, that compile - time dependencies have higher precedence than exports. Therefore if - you import a module and use its macros, it is a compile time dependency. + Cycle of length 3: - Runtime dependencies are added whenever you invoke another module - inside a function. Modules with runtime dependencies do not have - to be compiled when the callee changes, unless there is a transitive - compile or an outdated export time dependency between them. The option - `--label compile-connected` can be used to find the first case. + lib/c.ex + lib/b.ex (compile) + lib/a.ex + + This means `c.ex` depends on `b.ex` at compile time. Any compile dependency + in a cycle is by definition a compile-connected dependency, which must be + generally avoided, as explained earlier in the module documentation. ## Shared options @@ -846,44 +921,15 @@ defmodule Mix.Tasks.Xref do true -> file_references end - # Filter according to non direct label - file_references = filter(file_references, filter) - - # If a label is given, remove empty root nodes - file_references = - if opts[:label] do - for {_, [_ | _]} = pair <- file_references, into: %{}, do: pair - else - file_references - end - - roots = - if sources do - Enum.map(sources, &{&1, nil}) - else - file_references - |> Map.drop(sinks || []) - |> Enum.map(&{elem(&1, 0), nil}) - end - - callback = fn {file, type} -> - children = if opts[:only_nodes], do: [], else: Map.get(file_references, file, []) - type = type && "(#{type})" - {{file, type}, Enum.sort(children)} - end - {found, count} = case opts[:format] do "dot" -> + {roots, callback, count} = + roots_and_callback(file_references, filter, sources, sinks, opts) + path = Keyword.get(opts, :output, "xref_graph.dot") - Mix.Utils.write_dot_graph!( - path, - "xref graph", - Enum.sort(roots), - callback, - opts - ) + Mix.Utils.write_dot_graph!(path, "xref graph", Enum.sort(roots), callback, opts) if path != "-" do png_path = (path |> Path.rootname() |> Path.basename()) <> ".png" @@ -899,19 +945,22 @@ defmodule Mix.Tasks.Xref do |> Mix.shell().info() end - {:references, count_references(file_references)} + {:references, count} "stats" -> - print_stats(file_references, opts) + print_stats(file_references, filter, opts) {:stats, 0} "cycles" -> - {:cycles, print_cycles(file_references, opts)} + {:cycles, print_cycles(file_references, filter, opts)} other when other in [nil, "plain", "pretty"] -> + {roots, callback, count} = + roots_and_callback(file_references, filter, sources, sinks, opts) + Mix.Utils.print_tree(Enum.sort(roots), callback, opts) - {:references, count_references(file_references)} + {:references, count} other -> Mix.raise("Unknown --format #{other} in mix xref graph") @@ -920,28 +969,6 @@ defmodule Mix.Tasks.Xref do check_failure(found, count, opts[:fail_above]) end - defp count_references(file_references) do - Enum.reduce(file_references, 0, fn {_, refs}, total -> total + length(refs) end) - end - - defp filter_fn(file_references, :compile_connected), - do: fn {key, type} -> - type == :compile and match?([_ | _], file_references[key] || []) - end - - defp filter_fn(_file_references, filter), - do: fn {_key, type} -> type == filter end - - defp filter(file_references, :all), do: file_references - - defp filter(file_references, filter) do - filter_fn = filter_fn(file_references, filter) - - for {key, children} <- file_references, - into: %{}, - do: {key, Enum.filter(children, filter_fn)} - end - defp source_tree(file_references, keys) do keys |> Enum.reduce({%{}, %{}}, fn key, {acc, seen} -> @@ -980,7 +1007,59 @@ defmodule Mix.Tasks.Xref do end) end - defp print_stats(references, opts) do + defp roots_and_callback(file_references, filter, sources, sinks, opts) do + # Filter according to non direct label + file_references = transitive_filter(file_references, filter) + + # If a label is given, remove empty root nodes + file_references = + if opts[:label] do + for {_, [_ | _]} = pair <- file_references, into: %{}, do: pair + else + file_references + end + + roots = + if sources do + Enum.map(sources, &{&1, nil}) + else + file_references + |> Map.drop(sinks || []) + |> Enum.map(&{elem(&1, 0), nil}) + end + + callback = fn {file, type} -> + children = if opts[:only_nodes], do: [], else: Map.get(file_references, file, []) + type = type && "(#{type})" + {{file, type}, Enum.sort(children)} + end + + {roots, callback, count_references(file_references)} + end + + defp count_references(file_references) do + Enum.reduce(file_references, 0, fn {_, refs}, total -> total + length(refs) end) + end + + defp transitive_filter_fn(file_references, :compile_connected), + do: fn {key, type} -> + type == :compile and match?([_ | _], file_references[key] || []) + end + + defp transitive_filter_fn(_file_references, filter), + do: fn {_key, type} -> type == filter end + + defp transitive_filter(file_references, :all), do: file_references + + defp transitive_filter(file_references, filter) do + filter_fn = transitive_filter_fn(file_references, filter) + + for {key, children} <- file_references, + into: %{}, + do: {key, Enum.filter(children, filter_fn)} + end + + defp print_stats(references, filter, opts) do with_digraph(references, fn graph -> shell = Mix.shell() @@ -995,11 +1074,11 @@ defmodule Mix.Tasks.Xref do shell.info("Compile dependencies: #{counters.compile} (edges)") shell.info("Exports dependencies: #{counters.export} (edges)") shell.info("Runtime dependencies: #{counters.nil} (edges)") - shell.info("Cycles: #{length(cycles(graph, opts))}") + shell.info("Cycles: #{length(cycles(graph, filter, opts))}") outgoing = references - |> Enum.map(fn {file, _} -> {:digraph.out_degree(graph, file), file} end) + |> Enum.map(fn {file, _} -> {out_stats_filter(references, graph, file, filter), file} end) |> Enum.sort(:desc) |> Enum.take(10) @@ -1008,7 +1087,7 @@ defmodule Mix.Tasks.Xref do incoming = references - |> Enum.map(fn {file, _} -> {:digraph.in_degree(graph, file), file} end) + |> Enum.map(fn {file, _} -> {in_stats_filter(references, graph, file, filter), file} end) |> Enum.sort(:desc) |> Enum.take(10) @@ -1017,6 +1096,32 @@ defmodule Mix.Tasks.Xref do end) end + defp out_stats_filter(_references, graph, file, :all), do: :digraph.out_degree(graph, file) + + defp out_stats_filter(references, graph, file, filter) do + filter_fn = transitive_filter_fn(references, filter) + + graph + |> :digraph.out_neighbours(file) + |> Enum.count(fn v -> + {_edge, _v1, _v2, label} = :digraph.edge(graph, {file, v}) + filter_fn.({file, label}) + end) + end + + defp in_stats_filter(_references, graph, file, :all), do: :digraph.in_degree(graph, file) + + defp in_stats_filter(references, graph, file, filter) do + filter_fn = transitive_filter_fn(references, filter) + + graph + |> :digraph.in_neighbours(file) + |> Enum.count(fn v -> + {_edge, _v1, _v2, label} = :digraph.edge(graph, {v, file}) + filter_fn.({file, label}) + end) + end + defp with_digraph(references, callback) do graph = :digraph.new() @@ -1026,7 +1131,7 @@ defmodule Mix.Tasks.Xref do end for {file, deps} <- references, {dep, label} <- deps do - :digraph.add_edge(graph, file, dep, label) + :digraph.add_edge(graph, {file, dep}, file, dep, label) end callback.(graph) @@ -1035,7 +1140,7 @@ defmodule Mix.Tasks.Xref do end end - defp cycles(graph, opts) do + defp cycles(graph, filter, opts) do # Vertices order in cyclic_strong_components/1 return is arbitrary and changes between # OTP versions, sorting is necessary to make the output stable across versions. cycles = @@ -1044,21 +1149,47 @@ defmodule Mix.Tasks.Xref do |> Enum.reduce([], &inner_cycles(graph, Enum.sort(&1), &2)) |> Enum.map(&{length(&1), &1}) - if min = opts[:min_cycle_size], do: Enum.filter(cycles, &(elem(&1, 0) > min)), else: cycles + cycles = + if min = opts[:min_cycle_size] do + Enum.filter(cycles, &(elem(&1, 0) > min)) + else + cycles + end + + # :compile_connected is the same + if cycle_fn = cycle_filter_fn(filter) do + Enum.filter(cycles, fn {_length, cycle} -> Enum.any?(cycle, cycle_fn) end) + else + cycles + end end + # In cycles, a compile connected is compile + defp cycle_filter_fn(:all), do: nil + defp cycle_filter_fn(:compile_connected), do: cycle_filter_fn(:compile) + defp cycle_filter_fn(filter), do: fn {_node, type} -> type == filter end + defp inner_cycles(_graph, [], acc), do: acc defp inner_cycles(graph, [v | vertices], acc) do cycle = :digraph.get_cycle(graph, v) - inner_cycles(graph, vertices -- cycle, [cycle | acc]) + inner_cycles(graph, vertices -- cycle, [label_cycle(cycle, graph) | acc]) + end + + defp label_cycle([from, to | cycle], graph) do + {_edge, _v1, _v2, label} = :digraph.edge(graph, {from, to}) + [{to, label} | label_cycle([to | cycle], graph)] + end + + defp label_cycle([_from], _graph) do + [] end - defp print_cycles(references, opts) do + defp print_cycles(references, filter, opts) do with_digraph(references, fn graph -> shell = Mix.shell() - case graph |> cycles(opts) |> Enum.sort(:desc) do + case graph |> cycles(filter, opts) |> Enum.sort(:desc) do [] -> shell.info("No cycles found") 0 @@ -1069,8 +1200,9 @@ defmodule Mix.Tasks.Xref do for {length, cycle} <- cycles do shell.info("Cycle of length #{length}:\n") - for node <- cycle do - shell.info(" " <> node) + for {node, type} <- cycle do + type = if type, do: " (#{type})", else: "" + shell.info(" " <> node <> type) end shell.info("") diff --git a/lib/mix/test/mix/generator_test.exs b/lib/mix/test/mix/generator_test.exs index 5b7dcca6433..e393471628b 100644 --- a/lib/mix/test/mix/generator_test.exs +++ b/lib/mix/test/mix/generator_test.exs @@ -147,6 +147,13 @@ defmodule Mix.GeneratorTest do assert_received {:mix_shell, :yes?, ["foo already exists, overwrite?"]} end) end + + test "with `format_elixir: true`" do + in_tmp("create_file", fn -> + create_file("foo", "%{ foo: :bar }", format_elixir: true) + assert File.read!("foo") == "%{foo: :bar}\n" + end) + end end describe "copy_file/3" do diff --git a/lib/mix/test/mix/tasks/compile.elixir_test.exs b/lib/mix/test/mix/tasks/compile.elixir_test.exs index 48090929699..02dfbd96e8a 100644 --- a/lib/mix/test/mix/tasks/compile.elixir_test.exs +++ b/lib/mix/test/mix/tasks/compile.elixir_test.exs @@ -1597,11 +1597,7 @@ defmodule Mix.Tasks.Compile.ElixirTest do def foo(), do: B.foo() def bar(), do: B.bar() def __after_verify__(__MODULE__) do - if Code.ensure_loaded?(B) and not function_exported?(B, :foo, 0) do - :ok - else - IO.warn("AFTER_VERIFY", __ENV__) - end + IO.warn("AFTER_VERIFY", __ENV__) end end """) @@ -1630,10 +1626,9 @@ defmodule Mix.Tasks.Compile.ElixirTest do end) # Check B due to direct dependency on A - # Check C due to transient dependency on A assert output =~ "A.foo/0 is undefined or private" - assert output =~ "B.bar/0 is undefined or private" - assert output =~ "AFTER_VERIFY" + refute output =~ "B.bar/0 is undefined or private" + refute output =~ "AFTER_VERIFY" # Ensure only A was recompiled assert_received {:mix_shell, :info, ["Compiled lib/a.ex"]} @@ -1650,7 +1645,7 @@ defmodule Mix.Tasks.Compile.ElixirTest do assert output =~ "B.bar/0 is undefined or private" assert output =~ "AFTER_VERIFY" - # Now we change B and it must no longer emit an AFTER_VERIFY warning + # Now we change B and it must emit an AFTER_VERIFY warning File.write!("lib/b.ex", """ defmodule B do end @@ -1663,7 +1658,7 @@ defmodule Mix.Tasks.Compile.ElixirTest do assert output =~ "B.foo/0 is undefined or private" assert output =~ "B.bar/0 is undefined or private" - refute output =~ "AFTER_VERIFY" + assert output =~ "AFTER_VERIFY" end) end diff --git a/lib/mix/test/mix/tasks/deps.git_test.exs b/lib/mix/test/mix/tasks/deps.git_test.exs index 8ebca0d4744..cdaf7b337ca 100644 --- a/lib/mix/test/mix/tasks/deps.git_test.exs +++ b/lib/mix/test/mix/tasks/deps.git_test.exs @@ -530,43 +530,6 @@ defmodule Mix.Tasks.DepsGitTest do end) end - test "with ref" do - [last, _ | _] = get_git_repo_revs("git_repo") - - Process.put(:git_repo_opts, depth: 1, ref: last) - - in_fixture("no_mixfile", fn -> - Mix.Project.push(GitApp) - - Mix.Tasks.Deps.Get.run([]) - message = "* Getting git_repo (#{fixture_path("git_repo")} - #{last})" - assert_received {:mix_shell, :info, [^message]} - assert_shallow("deps/git_repo", 1) - end) - end - - test "changing refspec updates retaining depth" do - [last, first | _] = get_git_repo_revs("git_repo") - - Process.put(:git_repo_opts, ref: first, depth: 1) - - in_fixture("no_mixfile", fn -> - Mix.Project.push(GitApp) - - Mix.Tasks.Deps.Get.run([]) - message = "* Getting git_repo (#{fixture_path("git_repo")} - #{first})" - assert_received {:mix_shell, :info, [^message]} - assert_shallow("deps/git_repo", 1) - assert File.read!("mix.lock") =~ first - - # Change refspec - update_dep(ref: last, depth: 1) - Mix.Tasks.Deps.Get.run([]) - assert_shallow("deps/git_repo", 1) - assert File.read!("mix.lock") =~ last - end) - end - test "removing depth retains shallow repository" do # For compatibility and simplicity, we follow Git's behavior and do not # attempt to unshallow an existing repository. This should not be a diff --git a/lib/mix/test/mix/tasks/profile.tprof_test.exs b/lib/mix/test/mix/tasks/profile.tprof_test.exs new file mode 100644 index 00000000000..9494773ac77 --- /dev/null +++ b/lib/mix/test/mix/tasks/profile.tprof_test.exs @@ -0,0 +1,205 @@ +Code.require_file("../../test_helper.exs", __DIR__) + +defmodule Mix.Tasks.Profile.TprofTest do + use MixTest.Case + + import ExUnit.CaptureIO + alias Mix.Tasks.Profile.Tprof + + # TODO remove once we require Erlang/OTP 27+ + @moduletag skip: System.otp_release() < "27" + + @expr "Enum.each(1..5, &String.Chars.Integer.to_string/1)" + + test "profiles evaluated expression", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["-e", @expr]) + end) =~ ~r/String\.Chars\.Integer\.to_string\/1\s+\d/ + end) + end + + test "profiles evaluated expression in multiple processes", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["-e", "spawn(fn -> #{@expr} end)"]) + end) =~ ~r/String\.Chars\.Integer\.to_string\/1\s+\d/ + end) + end + + test "profiles the script", context do + in_tmp(context.test, fn -> + profile_script_name = "profile_script.ex" + File.write!(profile_script_name, @expr) + + assert capture_io(fn -> + Tprof.run([profile_script_name]) + end) =~ ~r/String\.Chars\.Integer\.to_string\/1\s+\d/ + end) + end + + test "filters based on calls count", context do + in_tmp(context.test, fn -> + result = + capture_io(fn -> + Tprof.run(["--calls", "5", "-e", @expr]) + end) + + assert result =~ "\nString.Chars.Integer.to_string/1" + refute result =~ "\nEnum.each/2" + end) + end + + test "filters based on time", context do + in_tmp(context.test, fn -> + result = + capture_io(fn -> + Tprof.run(["--time", "50", "-e", @expr]) + end) + + refute result =~ "\nEnum.each/2" + end) + end + + test "filters based on memory", context do + in_tmp(context.test, fn -> + result = + capture_io(fn -> + Tprof.run(["--type", "memory", "--memory", "10", "-e", @expr]) + end) + + assert result =~ "\n:erlang.integer_to_binary/1" + refute result =~ "\nEnum.each/2" + end) + end + + test "sorts based on calls count", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["--sort", "calls", "-e", @expr]) + end) =~ ~r/\nEnum\.each\/2.*\nString\.Chars\.Integer\.to_string\/1/s + end) + end + + test "sorts based on memory usage", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["--type", "memory", "--sort", "calls", "-e", @expr]) + end) =~ ~r/\nEnum\.each\/2.*\n:erlang\.integer_to_binary\/1/s + end) + end + + test "sorts based on memory per call", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["--type", "memory", "--sort", "per_call", "-e", @expr]) + end) =~ ~r/\n:erlang\.integer_to_binary\/1.*\nEnum\.each\/2/s + end) + end + + test "aggregates totals over all processes", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["--report", "total", "-e", @expr]) + end) =~ "Profile results over all processes" + end) + end + + test "Module matching", context do + in_tmp(context.test, fn -> + refute capture_io(fn -> + Tprof.run(["--matching", "Enum", "-e", @expr]) + end) =~ ~r/String\.Chars\.Integer\.to_string\/1/ + end) + end + + test "Module.function matching", context do + in_tmp(context.test, fn -> + refute capture_io(fn -> + Tprof.run(["--matching", "Enum.each", "-e", @expr]) + end) =~ ~r/anonymous fn\/3 in Enum\.each\/2/ + end) + end + + test "Module.function/arity matching", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["--matching", "Enum.each/8", "-e", @expr]) + end) =~ ~r/Profile done over 0 matching functions/ + end) + end + + test "errors on missing files", context do + in_tmp(context.test, fn -> + message = "No files matched pattern \"non-existent\" given to --require" + + assert_raise Mix.Error, message, fn -> + capture_io(fn -> Tprof.run(["-r", "non-existent"]) end) + end + + message = "No files matched pattern \"non-existent\" given to --require" + + assert_raise Mix.Error, message, fn -> + capture_io(fn -> Tprof.run(["-pr", "non-existent"]) end) + end + + assert_raise Mix.Error, "No such file: non-existent", fn -> + capture_io(fn -> Tprof.run(["non-existent"]) end) + end + + File.mkdir_p!("lib") + + assert_raise Mix.Error, "No such file: lib", fn -> + capture_io(fn -> Tprof.run(["lib"]) end) + end + end) + end + + test "warmup", context do + in_tmp(context.test, fn -> + assert capture_io(fn -> + Tprof.run(["-e", @expr]) + end) =~ "Warmup..." + + refute capture_io(fn -> + Tprof.run(["-e", @expr, "--no-warmup"]) + end) =~ "Warmup..." + end) + end + + test "errors on incompatible options", context do + in_tmp(context.test, fn -> + message = "Incompatible sort option :memory with type :time" + + assert_raise Mix.Error, message, fn -> + capture_io(fn -> Tprof.run(["-e", @expr, "--sort", "memory"]) end) + end + + message = "Incompatible sort option :time with type :calls" + + assert_raise Mix.Error, message, fn -> + capture_io(fn -> Tprof.run(["-e", @expr, "--type", "calls", "--sort", "time"]) end) + end + + message = "Incompatible use of memory option with type :time" + + assert_raise Mix.Error, message, fn -> + capture_io(fn -> Tprof.run(["-e", @expr, "--time", "1", "--memory", "2"]) end) + end + + message = "Incompatible use of time option with type :calls" + + assert_raise Mix.Error, message, fn -> + capture_io(fn -> Tprof.run(["-e", @expr, "--type", "calls", "--time", "1"]) end) + end + end) + end + + describe ".profile/2" do + test "returns the return value of the function call" do + capture_io(fn -> + assert 42 == Tprof.profile(fn -> 42 end) + end) + end + end +end diff --git a/lib/mix/test/mix/tasks/xref_test.exs b/lib/mix/test/mix/tasks/xref_test.exs index 51c2da7eabd..830fa083240 100644 --- a/lib/mix/test/mix/tasks/xref_test.exs +++ b/lib/mix/test/mix/tasks/xref_test.exs @@ -439,14 +439,85 @@ defmodule Mix.Tasks.XrefTest do """) end + test "stats with compile label" do + assert_graph(["--format", "stats", "--label", "compile"], """ + Tracked files: 5 (nodes) + Compile dependencies: 3 (edges) + Exports dependencies: 0 (edges) + Runtime dependencies: 3 (edges) + Cycles: 1 + + Top 5 files with most outgoing dependencies: + * lib/c.ex (1) + * lib/b.ex (1) + * lib/a.ex (1) + * lib/e.ex (0) + * lib/d.ex (0) + + Top 5 files with most incoming dependencies: + * lib/e.ex (1) + * lib/d.ex (1) + * lib/b.ex (1) + * lib/c.ex (0) + * lib/a.ex (0) + """) + end + + test "stats with compile-connected label" do + assert_graph(["--format", "stats", "--label", "compile-connected"], """ + Tracked files: 5 (nodes) + Compile dependencies: 3 (edges) + Exports dependencies: 0 (edges) + Runtime dependencies: 3 (edges) + Cycles: 1 + + Top 5 files with most outgoing dependencies: + * lib/c.ex (1) + * lib/b.ex (1) + * lib/a.ex (1) + * lib/e.ex (0) + * lib/d.ex (0) + + Top 5 files with most incoming dependencies: + * lib/d.ex (1) + * lib/b.ex (1) + * lib/e.ex (0) + * lib/c.ex (0) + * lib/a.ex (0) + """) + end + test "cycles" do assert_graph(["--format", "cycles"], """ 1 cycles found. Showing them in decreasing size: - Cycle of length 3: + Cycle of length 2: + + lib/b.ex (compile) + lib/a.ex + + """) + end + + test "cycles with compile label require at least one of such type" do + assert_graph(["--format", "cycles", "--label", "compile"], """ + 1 cycles found. Showing them in decreasing size: + + Cycle of length 2: + lib/b.ex (compile) lib/a.ex - lib/b.ex + + """) + end + + test "cycles with compile-connected label is the same as compile" do + assert_graph(["--format", "cycles", "--label", "compile-connected"], """ + 1 cycles found. Showing them in decreasing size: + + Cycle of length 2: + + lib/b.ex (compile) lib/a.ex """)