diff --git a/.sonarcloud.properties b/.sonarcloud.properties new file mode 100644 index 000000000..97e84857b --- /dev/null +++ b/.sonarcloud.properties @@ -0,0 +1,6 @@ +# Path to sources +sonar.sources=src +sonar.exclusions=src/UnitTests,src/TestResults + +# Path to tests +sonar.tests=src/UnitTests diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 89708bde3..4bcd6cc03 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,15 +4,15 @@ ### Prerequisites -1. .NET Core 2.2 SDK +1. .NET Core 3.1 SDK - [Windows](https://www.microsoft.com/net/learn/get-started/windows) - [Mac OS](https://www.microsoft.com/net/learn/get-started/macos) - [Linux](https://www.microsoft.com/net/learn/get-started/linux/rhel) 2. C# Extension to [VS Code](https://code.visualstudio.com) (all platforms) 3. Python 2.7 -4. Python 3.6 +4. Python 3.6+ -*Alternative:* [Visual Studio 2017 or 2019](https://www.visualstudio.com/downloads/) (Windows only) with .NET Core and C# Workloads. Community Edition is free and is fully functional. +*Alternative:* [Visual Studio 2019](https://www.visualstudio.com/downloads/) (Windows only) with .NET Core and C# Workloads. Community Edition is free and is fully functional. ### Setup @@ -54,7 +54,7 @@ On Windows you can also attach from Visual Studio (Debug | Attach To Process). ### Unit Tests To run unit tests, do one of the following: - Run the Unit Tests in the [VS Code Python Extension](https://github.com/Microsoft/vscode-python) project via *Launch Language Server Tests*. -- On Windows: open the `PLS.sln` solution in Visual Studio 2017 or 2019 and run tests from the Test Explorer. +- On Windows: open the `PLS.sln` solution in Visual Studio 2019 and run tests from the Test Explorer. - Run `dotnet test` from Terminal in the `src` directory, or in a specific directory like `src/Analysis/Ast/Test` to test a specific suite. - Install C# extension and .NET Core Test Explorer for VS Code, open src folder in VS Code and run tests. diff --git a/README.md b/README.md index ba346338a..dbc794445 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ The following diagnostics are supported: | `parameter-already-specified` | A argument with this name has already been specified. | | `parameter-missing` | A required positional argument is missing. | | `positional-argument-after-keyword` | A positional argument has been provided after a keyword argument. | +| `positional-only-named` | A positional-only argument (3.8+) has been named in a function call. | | `return-in-init` | Encountered an explicit return in `__init__` function. | | `typing-generic-arguments` | An error occurred while constructing `Generic`. | | `typing-newtype-arguments` | An error occurred while constructing `NewType`. | @@ -72,7 +73,7 @@ An example of a user configuration which sets these options: } ``` -Linting can also be controlled on an invidual line basis with a generalized `#noqa`. Lines with `#noqa` will have their diagnostic output suppressed. +Linting can also be controlled on an individual line basis with a generalized `#noqa`. Lines with `#noqa` will have their diagnostic output suppressed. An example usage: diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..8a5d128f3 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,41 @@ + + +## Security + +Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). + +If you believe you have found a security vulnerability in any Microsoft-owned repository that meets Microsoft's [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)) of a security vulnerability, please report it to us as described below. + +## Reporting Security Issues + +**Please do not report security vulnerabilities through public GitHub issues.** + +Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). + +If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). + +You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). + +Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: + + * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) + * Full paths of source file(s) related to the manifestation of the issue + * The location of the affected source code (tag/branch/commit or direct URL) + * Any special configuration required to reproduce the issue + * Step-by-step instructions to reproduce the issue + * Proof-of-concept or exploit code (if possible) + * Impact of the issue, including how an attacker might exploit the issue + +This information will help us triage your report more quickly. + +If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. + +## Preferred Languages + +We prefer all communications to be in English. + +## Policy + +Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). + + diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 7eccd4713..2181b3e58 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -86,6 +86,10 @@ fill in the rest of the information. Before the second pass is complete, some in will be incomplete, and some warnings about unresolved imports may occur. The analysis is complete when the status message (in the bottom bar) disappears. +### Completions do not update and/or import is still being reported as unresolved after `pip install` +Library search path watching is currently disabled by default in order to mitigate issues reported +in https://github.com/microsoft/python-language-server/pull/1841. If you require this feature, set +`"python.analysis.watchSearchPaths": true` in your user settings. ## Filing an issue diff --git a/azure-pipelines.yml b/azure-pipelines.yml new file mode 100644 index 000000000..3c993819d --- /dev/null +++ b/azure-pipelines.yml @@ -0,0 +1,45 @@ +# build number format +name: $(VersionMajorMinor)$(VersionPatchCounter) + +trigger: + branches: + include: + - master + - azure-pipelines + + +pool: + name: VSEngSS-MicroBuild2019-1ES + +# Need a job for win-x86,win-x64,osx-x64,linux-x64,any +jobs: + - job: x86 + displayName: "Build x86" + steps: + - template: build/azure-pipeline-steps.yml + parameters: + OSTarget: win-x86 + - job: x64 + displayName: "Build x64" + steps: + - template: build/azure-pipeline-steps.yml + parameters: + OSTarget: win-x64 + - job: DX64 + displayName: "Build osxX64" + steps: + - template: build/azure-pipeline-steps.yml + parameters: + OSTarget: osx-x64 + - job: LX64 + displayName: "Build linuxX64" + steps: + - template: build/azure-pipeline-steps.yml + parameters: + OSTarget: linux-x64 + - job: AnyCPU + displayName: "Build AnyCPU" + steps: + - template: build/azure-pipeline-steps.yml + parameters: + OSTarget: any diff --git a/build/NetStandard.settings b/build/NetStandard.settings index cb7c5ae9f..2d4474219 100644 --- a/build/NetStandard.settings +++ b/build/NetStandard.settings @@ -26,11 +26,6 @@ win-x86;win-x64;osx-x64;linux-x64 - - - true - - $(DefineConstants);SIGN true diff --git a/build/azure-pipeline-steps.yml b/build/azure-pipeline-steps.yml new file mode 100644 index 000000000..3504aec58 --- /dev/null +++ b/build/azure-pipeline-steps.yml @@ -0,0 +1,112 @@ +parameters: + name: OSTarget + type: string + default: 'win-x86' + +steps: + - task: MicroBuildSigningPlugin@3 + displayName: 'Install microbuild signing plugin' + condition: notin(variables['Build.Reason'], 'PullRequest') + inputs: + signType: 'Real' + zipSources: false + + - task: UseDotNet@2 + displayName: 'Use .Net Core SDK 3.1.x' + inputs: + version: 3.1.x + + - task: DotNetCoreCLI@2 + displayName: 'dotnet restore' + inputs: + command: restore + projects: | + src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj + src/Publish/SignLayout.csproj + restoreDirectory: '$(Build.BinariesDirectory)' + + - task: DotNetCoreCLI@2 + displayName: 'dotnet publish LanguageServer' + inputs: + command: publish + publishWebProjects: false + projects: src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj + arguments: '-c $(BuildConfiguration) -r ${{ parameters.OSTarget }} -o $(Build.BinariesDirectory)/out /p:SignType=None /p:Version=$(Build.BuildNumber)' + zipAfterPublish: false + modifyOutputPath: false + condition: and(succeeded(), ne('${{ parameters.OSTarget }}', 'any')) + + - task: DotNetCoreCLI@2 + displayName: 'dotnet publish LanguageServer for Any' + inputs: + command: publish + publishWebProjects: false + projects: src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj + arguments: '-c $(BuildConfiguration) -o $(Build.BinariesDirectory)/out /p:SignType=None /p:Version=$(Build.BuildNumber)' + zipAfterPublish: false + modifyOutputPath: false + condition: and(succeeded(), eq('${{ parameters.OSTarget }}', 'any')) + + - task: MSBuild@1 + displayName: 'Sign assemblies' + condition: notin(variables['Build.Reason'], 'PullRequest') + inputs: + solution: src/Publish/SignLayout.csproj + platform: '$(BuildPlatform)' + configuration: '$(BuildConfiguration)' + msbuildArguments: '/p:OutputPath=$(Build.BinariesDirectory)\out /p:SignServer=true /p:SignParser=true' + + - task: NuGetCommand@2 + displayName: 'NuGet pack language server' + inputs: + command: pack + packagesToPack: 'src/LanguageServer/Impl/Python-Language-Server.nuspec' + packDestination: '$(Build.ArtifactStagingDirectory)\packages' + versioningScheme: byEnvVar + versionEnvVar: NugetPackageVersion + buildProperties: 'os=-${{ parameters.OSTarget }}' + basePath: '$(Build.BinariesDirectory)/out' + + - task: NuGetCommand@2 + displayName: 'NuGet pack parsing' + inputs: + command: pack + packagesToPack: 'src/Parsing/Impl/Microsoft-Python-Parsing.nuspec' + packDestination: '$(Build.ArtifactStagingDirectory)\packages' + versioningScheme: byEnvVar + versionEnvVar: NugetPackageVersion + buildProperties: 'os=-${{ parameters.OSTarget }}' + basePath: '$(Build.BinariesDirectory)/out' + + - task: MSBuild@1 + displayName: 'Sign packages' + condition: notin(variables['Build.Reason'], 'PullRequest') + inputs: + solution: src/Publish/SignLayout.csproj + platform: '$(BuildPlatform)' + configuration: '$(BuildConfiguration)' + msbuildArguments: '/p:OutputPath=$(Build.ArtifactStagingDirectory)\packages /p:SignPackage=true /p:Version=$(Build.BuildNumber)' + + - task: PublishSymbols@2 + inputs: + SymbolsFolder: '$(Build.BinariesDirectory)' + SearchPattern: '**/out/**/*.pdb' + SymbolServerType: 'TeamServices' + condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'), eq('${{ parameters.OSTarget }}', 'any'), eq(variables['ShouldPublishNuget'], 'True'), eq(variables['SignType'], 'real')) + +# If API key expires, you need to generate a new one here: +# https://www.nuget.org/account/apikeys +# You'll need admin permission to associate the key with the Python Tools org + - task: NuGetCommand@2 + displayName: 'NuGet publish parsing ' + inputs: + command: push + packagesToPush: '$(Build.ArtifactStagingDirectory)/**/Microsoft.Python.Parsing*.nupkg;!$(Build.ArtifactStagingDirectory)/**/*.symbols.nupkg' + nuGetFeedType: external + publishFeedCredentials: 'python_language_server_nuget' + condition: and(succeeded(), eq(variables['ShouldPublishNuget'], 'True'), eq('${{ parameters.OSTarget }}', 'any'), notin(variables['Build.Reason'], 'PullRequest')) + timeoutInMinutes: 20 + + - task: MicroBuildCleanup@1 + displayName: 'Execute cleanup tasks' + condition: succeededOrFailed() diff --git a/src/Analysis/Ast/Impl/Analyzer/ActivityTracker.cs b/src/Analysis/Ast/Impl/Analyzer/ActivityTracker.cs index 937634006..e48c9bfb6 100644 --- a/src/Analysis/Ast/Impl/Analyzer/ActivityTracker.cs +++ b/src/Analysis/Ast/Impl/Analyzer/ActivityTracker.cs @@ -15,7 +15,6 @@ using System.Collections.Generic; using System.Diagnostics; -using System.Linq; namespace Microsoft.Python.Analysis.Analyzer { internal static class ActivityTracker { diff --git a/src/Analysis/Ast/Impl/Analyzer/AnalysisModuleKey.cs b/src/Analysis/Ast/Impl/Analyzer/AnalysisModuleKey.cs index 489ad4d72..a2d508cd0 100644 --- a/src/Analysis/Ast/Impl/Analyzer/AnalysisModuleKey.cs +++ b/src/Analysis/Ast/Impl/Analyzer/AnalysisModuleKey.cs @@ -22,20 +22,18 @@ namespace Microsoft.Python.Analysis.Analyzer { [DebuggerDisplay("{Name} : {FilePath}")] - internal readonly struct AnalysisModuleKey : IEquatable { + public readonly struct AnalysisModuleKey : IEquatable { public string Name { get; } public string FilePath { get; } public bool IsTypeshed { get; } - public bool IsNonUserAsDocument { get; } - public AnalysisModuleKey(IPythonModule module) { - Name = module.Name; - FilePath = module.ModuleType == ModuleType.CompiledBuiltin ? null : module.FilePath; - IsTypeshed = module is StubPythonModule stub && stub.IsTypeshed; - IsNonUserAsDocument = (module.IsNonUserFile() || module.IsCompiled()) && module is IDocument document && document.IsOpen; - } + public AnalysisModuleKey(IPythonModule module) : this( + module.Name, + module.ModuleType == ModuleType.CompiledBuiltin ? null : module.FilePath, + module.IsTypeshed, + IsNonUserAsDocumentModule(module)) { } - public AnalysisModuleKey(string name, string filePath, bool isTypeshed) + public AnalysisModuleKey(string name, string filePath, bool isTypeshed = false) : this(name, filePath, isTypeshed, false) { } private AnalysisModuleKey(string name, string filePath, bool isTypeshed, bool isNonUserAsDocument) { @@ -48,7 +46,7 @@ private AnalysisModuleKey(string name, string filePath, bool isTypeshed, bool is public AnalysisModuleKey GetNonUserAsDocumentKey() => new AnalysisModuleKey(Name, FilePath, IsTypeshed, true); public bool Equals(AnalysisModuleKey other) - => Name.EqualsOrdinal(other.Name) && FilePath.PathEquals(other.FilePath) && IsTypeshed == other.IsTypeshed && IsNonUserAsDocument == other.IsNonUserAsDocument; + => Name.EqualsOrdinal(other.Name) && FilePath.PathEquals(other.FilePath) && IsTypeshed == other.IsTypeshed; public override bool Equals(object obj) => obj is AnalysisModuleKey other && Equals(other); @@ -57,7 +55,6 @@ public override int GetHashCode() { var hashCode = Name != null ? Name.GetHashCode() : 0; hashCode = (hashCode * 397) ^ (FilePath != null ? FilePath.GetPathHashCode() : 0); hashCode = (hashCode * 397) ^ IsTypeshed.GetHashCode(); - hashCode = (hashCode * 397) ^ IsNonUserAsDocument.GetHashCode(); return hashCode; } } @@ -73,5 +70,10 @@ public void Deconstruct(out string moduleName, out string filePath, out bool isT } public override string ToString() => $"{Name}({FilePath})"; + + public bool IsNonUserAsDocument { get; } + + private static bool IsNonUserAsDocumentModule(IPythonModule module) + => (module.IsNonUserFile() || module.IsCompiled()) && module is IDocument document && document.IsOpen; } } diff --git a/src/Analysis/Ast/Impl/Analyzer/AnalysisWalker.cs b/src/Analysis/Ast/Impl/Analyzer/AnalysisWalker.cs index 8f5743c70..ed80d9018 100644 --- a/src/Analysis/Ast/Impl/Analyzer/AnalysisWalker.cs +++ b/src/Analysis/Ast/Impl/Analyzer/AnalysisWalker.cs @@ -18,6 +18,7 @@ using Microsoft.Python.Analysis.Analyzer.Evaluation; using Microsoft.Python.Analysis.Analyzer.Handlers; using Microsoft.Python.Analysis.Analyzer.Symbols; +using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Parsing.Ast; @@ -61,6 +62,26 @@ public override bool Walk(NamedExpression node) { return base.Walk(node); } + public override bool Walk(CallExpression node) { + Eval.ProcessCallForReferences(node); + return base.Walk(node); + } + + public override void PostWalk(DelStatement node) { + if (Module.ModuleType != ModuleType.User && + Eval.Services.GetService()?.Options.KeepLibraryAst != true) { + return; + } + + var names = node.Expressions.OfType() + .Concat(node.Expressions.OfType().SelectMany(t => t.Items.OfType())) + .Where(x => !string.IsNullOrEmpty(x.Name)); + + foreach (var nex in names) { + Eval.LookupNameInScopes(nex.Name)?.AddReference(Eval.GetLocationOfName(nex)); + } + } + public override bool Walk(ExpressionStatement node) { switch (node.Expression) { case ExpressionWithAnnotation ea: @@ -69,13 +90,6 @@ public override bool Walk(ExpressionStatement node) { case Comprehension comp: Eval.ProcessComprehension(comp); return false; - case CallExpression callex when callex.Target is NameExpression nex && !string.IsNullOrEmpty(nex.Name): - Eval.LookupNameInScopes(nex.Name)?.AddReference(Eval.GetLocationOfName(nex)); - return true; - case CallExpression callex when callex.Target is MemberExpression mex && !string.IsNullOrEmpty(mex.Name): - var t = Eval.GetValueFromExpression(mex.Target)?.GetPythonType(); - t?.GetMember(mex.Name)?.AddReference(Eval.GetLocationOfName(mex)); - return true; default: return base.Walk(node); } @@ -89,18 +103,12 @@ public override bool Walk(ExpressionStatement node) { public override bool Walk(ImportStatement node) => ImportHandler.HandleImport(node); public override bool Walk(NonlocalStatement node) => NonLocalHandler.HandleNonLocal(node); - public override bool Walk(TryStatement node) { - TryExceptHandler.HandleTryExcept(node); - return base.Walk(node); - } + public override bool Walk(TryStatement node) => TryExceptHandler.HandleTryExcept(node); - public override bool Walk(WhileStatement node) { - LoopHandler.HandleWhile(node); - return base.Walk(node); - } + public override bool Walk(WhileStatement node) => LoopHandler.HandleWhile(node); public override bool Walk(WithStatement node) { - WithHandler.HandleWith(node); + WithHandler.HandleWith(node); // HandleWith does not walk the body. return base.Walk(node); } #endregion diff --git a/src/Analysis/Ast/Impl/Analyzer/CachedAnalysis.cs b/src/Analysis/Ast/Impl/Analyzer/CachedAnalysis.cs new file mode 100644 index 000000000..290c592d8 --- /dev/null +++ b/src/Analysis/Ast/Impl/Analyzer/CachedAnalysis.cs @@ -0,0 +1,80 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis.Analyzer.Evaluation; +using Microsoft.Python.Analysis.Diagnostics; +using Microsoft.Python.Analysis.Documents; +using Microsoft.Python.Analysis.Utilities; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core; +using Microsoft.Python.Core.Diagnostics; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Analyzer { + /// + /// Analysis of a module restored from database. + /// + internal sealed class CachedAnalysis : IDocumentAnalysis { + public CachedAnalysis(IDocument document, IServiceContainer services) { + Check.ArgumentNotNull(nameof(document), document); + Document = document; + ExpressionEvaluator = new ExpressionEval(services, document, AstUtilities.MakeEmptyAst(document.Uri)); + } + + #region IDocumentAnalysis + /// + /// Analyzed document. + /// + public IDocument Document { get; } + + /// + /// Version of the analysis. Usually matches document version, + /// but can be lower when document or its dependencies were + /// updated since. + /// + public int Version => 0; + + /// + /// Empty AST. + /// + public PythonAst Ast => ExpressionEvaluator.Ast; + + /// + /// Document/module global scope. + /// + public IGlobalScope GlobalScope => Document.GlobalScope; + + /// + /// Expression evaluator used in the analysis. + /// Only supports scope operation since there is no AST + /// when library analysis is complete. + /// + public IExpressionEvaluator ExpressionEvaluator { get; } + + /// + /// Members of the module which are transferred during a star import. null means __all__ was not defined. + /// + public IReadOnlyList StarImportMemberNames => Array.Empty(); + + /// + /// Analysis diagnostics. + /// + public IEnumerable Diagnostics => Enumerable.Empty(); + #endregion + } +} diff --git a/src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs b/src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs index 2b9182661..b13891861 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs @@ -14,6 +14,7 @@ // permissions and limitations under the License. using Microsoft.Python.Analysis.Dependencies; +using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.Analysis.Analyzer { /// @@ -30,6 +31,11 @@ internal interface IAnalyzable { /// void NotifyAnalysisBegins(); + /// + /// Performs standard analysis pass. Does not include any restoration from databases. + /// + ModuleWalker Analyze(PythonAst ast); + /// /// Notifies document that its analysis is now complete. /// diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Annotations.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Annotations.cs index d0b3d5747..30b8e3fb8 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Annotations.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Annotations.cs @@ -19,17 +19,17 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation { internal sealed partial class ExpressionEval { - public IPythonType GetTypeFromAnnotation(Expression expr, LookupOptions options = LookupOptions.Global | LookupOptions.Builtins) + public IPythonType GetTypeFromAnnotation(Expression expr, LookupOptions options = LookupOptions.Normal) => GetTypeFromAnnotation(expr, out _, options); - public IPythonType GetTypeFromAnnotation(Expression expr, out bool isGeneric, LookupOptions options = LookupOptions.Global | LookupOptions.Builtins) { + public IPythonType GetTypeFromAnnotation(Expression expr, out bool isGeneric, LookupOptions lookupOptions = LookupOptions.Normal) { isGeneric = false; switch (expr) { case null: return null; case NameExpression nameExpr: // x: T - var name = GetValueFromExpression(nameExpr); + var name = GetValueFromExpression(nameExpr, lookupOptions); if(name is IGenericTypeParameter gtp) { isGeneric = true; return gtp; @@ -37,11 +37,11 @@ public IPythonType GetTypeFromAnnotation(Expression expr, out bool isGeneric, Lo break; case CallExpression callExpr: // x: NamedTuple(...) - return GetValueFromCallable(callExpr)?.GetPythonType() ?? UnknownType; + return GetValueFromCallable(callExpr, lookupOptions)?.GetPythonType() ?? UnknownType; case IndexExpression indexExpr: // Try generics - var target = GetValueFromExpression(indexExpr.Target); - var result = GetValueFromGeneric(target, indexExpr); + var target = GetValueFromExpression(indexExpr.Target, lookupOptions); + var result = GetValueFromGeneric(target, indexExpr, lookupOptions); if (result != null) { isGeneric = true; return result.GetPythonType(); @@ -51,7 +51,7 @@ public IPythonType GetTypeFromAnnotation(Expression expr, out bool isGeneric, Lo // Look at specialization and typing first var ann = new TypeAnnotation(Ast.LanguageVersion, expr); - return ann.GetValue(new TypeAnnotationConverter(this, expr, options)); + return ann.GetValue(new TypeAnnotationConverter(this, expr, lookupOptions)); } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs index 098595dde..436adc1f9 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs @@ -27,15 +27,15 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation { internal sealed partial class ExpressionEval { private readonly Stack _callEvalStack = new Stack(); - public IMember GetValueFromCallable(CallExpression expr) { + public IMember GetValueFromCallable(CallExpression expr, LookupOptions lookupOptions = LookupOptions.Normal) { if (expr?.Target == null) { return null; } - var target = GetValueFromExpression(expr.Target); + var target = GetValueFromExpression(expr.Target, lookupOptions); target?.AddReference(GetLocationOfName(expr.Target)); - var result = GetValueFromGeneric(target, expr); + var result = GetValueFromGeneric(target, expr, lookupOptions); if (result != null) { return result; } @@ -53,7 +53,7 @@ public IMember GetValueFromCallable(CallExpression expr) { value = GetValueFromInstanceCall(pi, expr); break; case IPythonFunctionType ft: // Standalone function or a class method call. - var instance = ft.DeclaringType?.CreateInstance(args); + var instance = ft.DeclaringType?.CreateInstance(args) as IPythonInstance; value = GetValueFromFunctionType(ft, instance, expr); break; case IPythonClassType cls: @@ -316,7 +316,11 @@ private void LoadFunctionDependencyModules(IPythonFunctionType fn) { } } - public IReadOnlyList CreateFunctionParameters(IPythonClassType self, IPythonClassMember function, FunctionDefinition fd, bool declareVariables) { + public IReadOnlyList CreateFunctionParameters( + IPythonClassType self, + IPythonClassMember function, + FunctionDefinition fd, + bool declareVariables) { // For class method no need to add extra parameters, but first parameter type should be the class. // For static and unbound methods do not add or set anything. // For regular bound methods add first parameter and set it to the class. @@ -353,7 +357,7 @@ public IReadOnlyList CreateFunctionParameters(IPythonClassType s // since outer type may be getting redefined. Consider 's = None; def f(s: s = 123): ... paramType = GetTypeFromAnnotation(p.Annotation, out isGeneric, LookupOptions.Local | LookupOptions.Builtins); // Default value of None does not mean the parameter is None, just says it can be missing. - defaultValue = defaultValue.IsUnknown() || defaultValue.IsOfType(BuiltinTypeId.NoneType) ? null : defaultValue; + defaultValue = defaultValue.IsUnknown() || defaultValue.IsOfType(BuiltinTypeId.None) ? null : defaultValue; if (paramType == null && defaultValue != null) { paramType = defaultValue.GetPythonType(); } @@ -383,5 +387,28 @@ private void DeclareParameter(Parameter p, ParameterInfo pi) { DeclareVariable(p.Name, paramType.CreateInstance(ArgumentSet.Empty(p.NameExpression, this)), VariableSource.Declaration, p.NameExpression); } + + internal void ProcessCallForReferences(CallExpression callExpr, LookupOptions lookupOptions = LookupOptions.Normal) { + if (Module.ModuleType != ModuleType.User && + Services.GetService()?.Options.KeepLibraryAst != true) { + return; + } + + switch (callExpr.Target) { + case NameExpression nex when !string.IsNullOrEmpty(nex.Name): + // Add reference to the function + this.LookupNameInScopes(nex.Name, lookupOptions)?.AddReference(GetLocationOfName(nex)); + break; + case MemberExpression mex when !string.IsNullOrEmpty(mex.Name): + var t = GetValueFromExpression(mex.Target, lookupOptions)?.GetPythonType(); + t?.GetMember(mex.Name)?.AddReference(GetLocationOfName(mex)); + break; + } + + // Add references to all arguments. + foreach (var arg in callExpr.Args) { + GetValueFromExpression(arg.Expression, lookupOptions); + } + } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Collections.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Collections.cs index 8a4b13f36..4e984d9c9 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Collections.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Collections.cs @@ -25,14 +25,14 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation { internal sealed partial class ExpressionEval { private const int MaxCollectionSize = 1000; - public IMember GetValueFromIndex(IndexExpression expr) { + public IMember GetValueFromIndex(IndexExpression expr, LookupOptions lookupOptions = LookupOptions.Normal) { if (expr?.Target == null) { return null; } - var target = GetValueFromExpression(expr.Target); + var target = GetValueFromExpression(expr.Target, lookupOptions); // Try generics first since this may be an expression like Dict[int, str] - var result = GetValueFromGeneric(target, expr); + var result = GetValueFromGeneric(target, expr, lookupOptions); if (result != null) { return result; } @@ -45,63 +45,65 @@ public IMember GetValueFromIndex(IndexExpression expr) { var type = target.GetPythonType(); if (type != null) { if (!(target is IPythonInstance instance)) { - instance = type.CreateInstance(ArgumentSet.Empty(expr, this)); + instance = type.CreateInstance(ArgumentSet.Empty(expr, this)) as IPythonInstance; } - var index = GetValueFromExpression(expr.Index); - if (index != null) { - return type.Index(instance, new ArgumentSet(new[] { index }, expr, this)); + if (instance != null) { + var index = GetValueFromExpression(expr.Index, lookupOptions); + if (index != null) { + return type.Index(instance, new ArgumentSet(new[] {index}, expr, this)); + } } } return UnknownType; } - public IMember GetValueFromList(ListExpression expression) { + public IMember GetValueFromList(ListExpression expression, LookupOptions lookupOptions = LookupOptions.Normal) { var contents = new List(); foreach (var item in expression.Items.Take(MaxCollectionSize)) { - var value = GetValueFromExpression(item) ?? UnknownType; + var value = GetValueFromExpression(item, lookupOptions) ?? UnknownType; contents.Add(value); } return PythonCollectionType.CreateList(Module, contents, exact: expression.Items.Count <= MaxCollectionSize); } - public IMember GetValueFromDictionary(DictionaryExpression expression) { + public IMember GetValueFromDictionary(DictionaryExpression expression, LookupOptions lookupOptions = LookupOptions.Normal) { var contents = new Dictionary(); foreach (var item in expression.Items.Take(MaxCollectionSize)) { - var key = GetValueFromExpression(item.SliceStart) ?? UnknownType; - var value = GetValueFromExpression(item.SliceStop) ?? UnknownType; + var key = GetValueFromExpression(item.SliceStart, lookupOptions) ?? UnknownType; + var value = GetValueFromExpression(item.SliceStop, lookupOptions) ?? UnknownType; contents[key] = value; } return new PythonDictionary(Module, contents, exact: expression.Items.Count <= MaxCollectionSize); } - private IMember GetValueFromTuple(TupleExpression expression) { + private IMember GetValueFromTuple(TupleExpression expression, LookupOptions lookupOptions = LookupOptions.Normal) { var contents = new List(); foreach (var item in expression.Items.Take(MaxCollectionSize)) { - var value = GetValueFromExpression(item) ?? UnknownType; + var value = GetValueFromExpression(item, lookupOptions) ?? UnknownType; contents.Add(value); } return PythonCollectionType.CreateTuple(Module, contents, exact: expression.Items.Count <= MaxCollectionSize); } - public IMember GetValueFromSet(SetExpression expression) { + public IMember GetValueFromSet(SetExpression expression, LookupOptions lookupOptions = LookupOptions.Normal) { var contents = new List(); foreach (var item in expression.Items.Take(MaxCollectionSize)) { - var value = GetValueFromExpression(item) ?? UnknownType; + var value = GetValueFromExpression(item, lookupOptions) ?? UnknownType; contents.Add(value); } return PythonCollectionType.CreateSet(Module, contents, exact: expression.Items.Count <= MaxCollectionSize); } - public IMember GetValueFromGenerator(GeneratorExpression expression) { + public IMember GetValueFromGenerator(GeneratorExpression expression, LookupOptions lookupOptions = LookupOptions.Normal) { var iter = expression.Iterators.OfType().FirstOrDefault(); if (iter != null) { - return GetValueFromExpression(iter.List) ?? UnknownType; + return GetValueFromExpression(iter.List, lookupOptions) ?? UnknownType; } return UnknownType; } - public IMember GetValueFromComprehension(Comprehension node) { + public IMember GetValueFromComprehension(Comprehension node, LookupOptions lookupOptions = LookupOptions.Normal) { var oldVariables = CurrentScope.Variables.OfType().ToDictionary(k => k.Name, v => v); try { ProcessComprehension(node); @@ -109,14 +111,14 @@ public IMember GetValueFromComprehension(Comprehension node) { // TODO: Evaluate comprehensions to produce exact contents, if possible. switch (node) { case ListComprehension lc: - var v1 = GetValueFromExpression(lc.Item) ?? UnknownType; + var v1 = GetValueFromExpression(lc.Item, lookupOptions) ?? UnknownType; return PythonCollectionType.CreateList(Module, new[] { v1 }); case SetComprehension sc: - var v2 = GetValueFromExpression(sc.Item) ?? UnknownType; + var v2 = GetValueFromExpression(sc.Item, lookupOptions) ?? UnknownType; return PythonCollectionType.CreateSet(Module, new[] { v2 }); case DictionaryComprehension dc: - var k = GetValueFromExpression(dc.Key) ?? UnknownType; - var v = GetValueFromExpression(dc.Value) ?? UnknownType; + var k = GetValueFromExpression(dc.Key, lookupOptions) ?? UnknownType; + var v = GetValueFromExpression(dc.Value, lookupOptions) ?? UnknownType; return new PythonDictionary(new PythonDictionaryType(Interpreter.ModuleResolution.BuiltinsModule), new Dictionary { { k, v } }); } @@ -134,9 +136,9 @@ public IMember GetValueFromComprehension(Comprehension node) { } } - internal void ProcessComprehension(Comprehension node) { + internal void ProcessComprehension(Comprehension node, LookupOptions lookupOptions = LookupOptions.Normal) { foreach (var cfor in node.Iterators.OfType().Where(c => c.Left != null)) { - var value = GetValueFromExpression(cfor.List); + var value = GetValueFromExpression(cfor.List, lookupOptions); if (value != null) { switch (cfor.Left) { case NameExpression nex when value is IPythonCollection c1: diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Constants.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Constants.cs index 37a2c7403..e55dd0ecd 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Constants.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Constants.cs @@ -23,7 +23,7 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation { internal sealed partial class ExpressionEval { - public IPythonInstance GetConstantFromLiteral(Expression expr) { + public IMember GetConstantFromLiteral(Expression expr) { if (expr is ConstantExpression ce) { switch (ce.Value) { case string s: @@ -44,7 +44,7 @@ public IPythonInstance GetConstantFromLiteral(Expression expr) { public IPythonType GetTypeFromLiteral(Expression expr) { if (expr is ConstantExpression ce) { if (ce.Value == null) { - return Interpreter.GetBuiltinType(BuiltinTypeId.NoneType); + return Interpreter.GetBuiltinType(BuiltinTypeId.None); } switch (Type.GetTypeCode(ce.Value.GetType())) { diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs index 7a70a01cb..966d83e70 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs @@ -34,7 +34,7 @@ internal sealed partial class ExpressionEval { /// and the specific type arguments, such as Generic[T] or constructor /// of a generic class. /// - private IMember GetValueFromGeneric(IMember target, Expression expr) { + private IMember GetValueFromGeneric(IMember target, Expression expr, LookupOptions lookupOptions) { if (!(target is IGenericType t && t.IsGeneric())) { return null; } @@ -48,13 +48,13 @@ private IMember GetValueFromGeneric(IMember target, Expression expr) { // Indexing returns type as from A[int] case IndexExpression indexExpr when target is IGenericType gt: // Generic[T1, T2, ...] - var indices = EvaluateIndex(indexExpr); + var indices = EvaluateIndex(indexExpr, lookupOptions); return CreateSpecificTypeFromIndex(gt, indices, expr); case CallExpression callExpr when target is PythonClassType c1: // Alternative instantiation: // class A(Generic[T]): ... // x = A(1234) - var arguments = EvaluateCallArgs(callExpr).ToArray(); + var arguments = EvaluateCallArgs(callExpr, lookupOptions).ToArray(); return CreateClassInstance(c1, arguments, callExpr); } } @@ -111,17 +111,17 @@ private IMember CreateSpecificTypeFromIndex(IGenericType gt, IReadOnlyList 0 ? gt.CreateSpecificType(new ArgumentSet(args, expr, this)) : UnknownType; } - private IReadOnlyList EvaluateIndex(IndexExpression expr) { + private IReadOnlyList EvaluateIndex(IndexExpression expr, LookupOptions lookupOptions) { var indices = new List(); if (expr.Index is TupleExpression tex) { foreach (var item in tex.Items) { - var e = GetValueFromExpression(item); - var forwardRef = GetValueFromForwardRef(e); + var e = GetValueFromExpression(item, lookupOptions); + var forwardRef = GetValueFromForwardRef(e, lookupOptions); indices.Add(forwardRef ?? e); } } else { - var index = GetValueFromExpression(expr.Index); - var forwardRef = GetValueFromForwardRef(index); + var index = GetValueFromExpression(expr.Index, lookupOptions); + var forwardRef = GetValueFromForwardRef(index, lookupOptions); if (forwardRef != null) { indices.Add(forwardRef); @@ -140,20 +140,20 @@ private IReadOnlyList EvaluateIndex(IndexExpression expr) { /// List['str'] => List[str] /// 'A[int]' => A[int] /// - private IMember GetValueFromForwardRef(IMember index) { + private IMember GetValueFromForwardRef(IMember index, LookupOptions lookupOptions) { index.TryGetConstant(out string forwardRefStr); if (string.IsNullOrEmpty(forwardRefStr)) { return null; } var forwardRefExpr = AstUtilities.TryCreateExpression(forwardRefStr, Interpreter.LanguageVersion); - return GetValueFromExpression(forwardRefExpr); + return GetValueFromExpression(forwardRefExpr, lookupOptions); } - private IReadOnlyList EvaluateCallArgs(CallExpression expr) { + private IReadOnlyList EvaluateCallArgs(CallExpression expr, LookupOptions lookupOptions = LookupOptions.Normal) { var indices = new List(); foreach (var e in expr.Args.Select(a => a.Expression)) { - var value = GetValueFromExpression(e) ?? UnknownType; + var value = GetValueFromExpression(e, lookupOptions) ?? UnknownType; indices.Add(value); } return indices; diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Operators.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Operators.cs index a21af69f9..dbd3f3116 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Operators.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Operators.cs @@ -22,7 +22,7 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation { internal sealed partial class ExpressionEval { - private IMember GetValueFromUnaryOp(UnaryExpression expr) { + private IMember GetValueFromUnaryOp(UnaryExpression expr, LookupOptions lookupOptions) { switch (expr.Op) { case PythonOperator.Not: case PythonOperator.Is: @@ -31,17 +31,17 @@ private IMember GetValueFromUnaryOp(UnaryExpression expr) { return Interpreter.GetBuiltinType(BuiltinTypeId.Bool); case PythonOperator.Invert: - return GetValueFromUnaryOp(expr, "__invert__"); + return GetValueFromUnaryOp(expr, "__invert__", lookupOptions); case PythonOperator.Negate: - return GetValueFromUnaryOp(expr, "__neg__"); + return GetValueFromUnaryOp(expr, "__neg__", lookupOptions); case PythonOperator.Pos: - return GetValueFromUnaryOp(expr, "__pos__"); + return GetValueFromUnaryOp(expr, "__pos__", lookupOptions); } return UnknownType; } - private IMember GetValueFromUnaryOp(UnaryExpression expr, string op) { - var target = GetValueFromExpression(expr.Expression); + private IMember GetValueFromUnaryOp(UnaryExpression expr, string op, LookupOptions lookupOptions) { + var target = GetValueFromExpression(expr.Expression, lookupOptions); if (target is IPythonInstance instance) { var fn = instance.GetPythonType()?.GetMember(op); // Process functions declared in code modules. Scraped/compiled/stub modules do not actually perform any operations. @@ -58,20 +58,21 @@ private IMember GetValueFromUnaryOp(UnaryExpression expr, string op) { return UnknownType; } - private IMember GetValueFromBinaryOp(Expression expr) { + private IMember GetValueFromBinaryOp(Expression expr, LookupOptions lookupOptions) { if (expr is AndExpression a) { - GetValueFromExpression(a.Left); - GetValueFromExpression(a.Right); + GetValueFromExpression(a.Left, lookupOptions); + GetValueFromExpression(a.Right, lookupOptions); return Interpreter.GetBuiltinType(BuiltinTypeId.Bool); } if (expr is OrExpression orexp) { // Consider 'self.__params = types.MappingProxyType(params or {})' - var leftSide = GetValueFromExpression(orexp.Left); + var leftSide = GetValueFromExpression(orexp.Left, lookupOptions); + // Do evaluate both sides in order to correctly track references + var rightSide = GetValueFromExpression(orexp.Right, lookupOptions); if (!leftSide.IsUnknown()) { return leftSide; } - var rightSide = GetValueFromExpression(orexp.Right); return rightSide.IsUnknown() ? Interpreter.GetBuiltinType(BuiltinTypeId.Bool) : rightSide; } @@ -81,18 +82,21 @@ private IMember GetValueFromBinaryOp(Expression expr) { var op = binop.Operator; - var left = GetValueFromExpression(binop.Left) ?? UnknownType; - var right = GetValueFromExpression(binop.Right) ?? UnknownType; + var left = GetValueFromExpression(binop.Left, lookupOptions) ?? UnknownType; + var right = GetValueFromExpression(binop.Right, lookupOptions) ?? UnknownType; if (left.IsUnknown() && right.IsUnknown()) { // Fast path for when nothing below will give any results. if (op.IsComparison()) { return Interpreter.GetBuiltinType(BuiltinTypeId.Bool); } - return UnknownType; } + var leftValue = left is IVariable v1 ? v1.Value : left; + var rightValue = right is IVariable v2 ? v2.Value : right; + var isInstance = leftValue is IPythonInstance || rightValue is IPythonInstance; + var leftType = left.GetPythonType(); var rightType = right.GetPythonType(); @@ -121,7 +125,7 @@ private IMember GetValueFromBinaryOp(Expression expr) { if (leftIsSupported && rightIsSupported) { if (TryGetValueFromBuiltinBinaryOp(op, leftTypeId, rightTypeId, Interpreter.LanguageVersion.Is3x(), out var member)) { - return member; + return isInstance ? new PythonInstance(member.GetPythonType()) : member; } } @@ -136,11 +140,10 @@ private IMember GetValueFromBinaryOp(Expression expr) { ret = CallOperator(op, left, leftType, right, rightType, expr, tryLeft: false); } - if (!ret.IsUnknown()) { - return ret; + if (ret.IsUnknown()) { + ret = op.IsComparison() ? Interpreter.GetBuiltinType(BuiltinTypeId.Bool) : left; } - - return op.IsComparison() ? Interpreter.GetBuiltinType(BuiltinTypeId.Bool) : left; + return isInstance ? new PythonInstance(ret.GetPythonType()) : ret; } if (rightIsSupported) { diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Scopes.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Scopes.cs index 188933339..c06d04bc2 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Scopes.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Scopes.cs @@ -21,7 +21,6 @@ using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core; using Microsoft.Python.Core.Disposables; -using Microsoft.Python.Core.Text; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.Analysis.Analyzer.Evaluation { @@ -40,9 +39,6 @@ public T GetInScope(string name, IScope scope) where T : class, IMember public void DeclareVariable(string name, IMember value, VariableSource source) => DeclareVariable(name, value, source, default(Location)); - public void DeclareVariable(string name, IMember value, VariableSource source, IPythonModule module) - => DeclareVariable(name, value, source, new Location(module)); - public void DeclareVariable(string name, IMember value, VariableSource source, Node location, bool overwrite = true) => DeclareVariable(name, value, source, GetLocationOfName(location), overwrite); @@ -58,12 +54,12 @@ public void DeclareVariable(string name, IMember value, VariableSource source, L if (member != null && !overwrite) { return; } - + if (source == VariableSource.Import && value is IVariable v) { CurrentScope.LinkVariable(name, v, location); return; } - + if (member != null) { if (!value.IsUnknown()) { CurrentScope.DeclareVariable(name, value, source, location); @@ -166,7 +162,7 @@ public IDisposable OpenScope(IPythonModule module, ScopeStatement node, out Scop // node points to global scope, it is not a function or a class. scope = gs; } else { - scope = outerScope.Children.OfType().FirstOrDefault(s => s.Node == node); + scope = outerScope.GetChildScope(node) as Scope; if (scope == null) { scope = new Scope(node, outerScope, Module); outerScope.AddChildScope(scope); @@ -180,6 +176,8 @@ public IDisposable OpenScope(IPythonModule module, ScopeStatement node, out Scop return new ScopeTracker(this); } + internal void ReplaceVariable(IVariable v) => CurrentScope.ReplaceVariable(v); + private class ScopeTracker : IDisposable { private readonly ExpressionEval _eval; @@ -194,10 +192,17 @@ public void Dispose() { // them better. // TODO: figure out threading/locking for the Open/Close pairs. // Debug.Assert(_eval._openScopes.Count > 0, "Attempt to close global scope"); - if (_eval._openScopes.Count > 0) { - _eval._openScopes.Pop(); + try { + if (_eval._openScopes.Count > 0) { + _eval._openScopes.Pop(); + } + _eval.CurrentScope = _eval._openScopes.Count == 0 ? _eval.GlobalScope : _eval._openScopes.Peek(); + } catch (InvalidOperationException) { + // Per comment above this can happen occasionally. + // The catch is tactical fix to prevent crashes since complete handling of open/close + // in threaded cases would be much larger change. + _eval.Log?.Log(TraceEventType.Verbose, "Error: Mismatched open/close in scope tracker - scope stack is empty on Dispose()"); } - _eval.CurrentScope = _eval._openScopes.Count == 0 ? _eval.GlobalScope : _eval._openScopes.Peek(); } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs index 1136a9091..03a37a1c2 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs @@ -60,7 +60,9 @@ public ExpressionEval(IServiceContainer services, IPythonModule module, PythonAs public LocationInfo GetLocationInfo(Node node) => node?.GetLocation(this) ?? LocationInfo.Empty; public Location GetLocationOfName(Node node) { - if (node == null || (Module.ModuleType != ModuleType.User && Module.ModuleType != ModuleType.Library)) { + if (node == null || + Module.ModuleType == ModuleType.Specialized || Module.ModuleType == ModuleType.Compiled || + Module.ModuleType == ModuleType.CompiledBuiltin || Module.ModuleType == ModuleType.Builtins) { return DefaultLocation; } @@ -124,7 +126,7 @@ public IDisposable OpenScope(IScope scope) { public IDisposable OpenScope(IPythonModule module, ScopeStatement scope) => OpenScope(module, scope, out _); #endregion - public IMember GetValueFromExpression(Expression expr, LookupOptions options = LookupOptions.Normal) { + public IMember GetValueFromExpression(Expression expr, LookupOptions lookupOptions = LookupOptions.Normal) { if (expr == null) { return null; } @@ -134,43 +136,43 @@ public IMember GetValueFromExpression(Expression expr, LookupOptions options = L IMember m; switch (expr) { case NameExpression nex: - m = GetValueFromName(nex, options); + m = GetValueFromName(nex, lookupOptions); break; case MemberExpression mex: - m = GetValueFromMember(mex); + m = GetValueFromMember(mex, lookupOptions); break; case CallExpression cex: - m = GetValueFromCallable(cex); + m = GetValueFromCallable(cex, lookupOptions); break; case UnaryExpression uex: - m = GetValueFromUnaryOp(uex); + m = GetValueFromUnaryOp(uex, lookupOptions); break; case IndexExpression iex: - m = GetValueFromIndex(iex); + m = GetValueFromIndex(iex, lookupOptions); break; case ConditionalExpression coex: - m = GetValueFromConditional(coex); + m = GetValueFromConditional(coex, lookupOptions); break; case ListExpression listex: - m = GetValueFromList(listex); + m = GetValueFromList(listex, lookupOptions); break; case DictionaryExpression dictex: - m = GetValueFromDictionary(dictex); + m = GetValueFromDictionary(dictex, lookupOptions); break; case SetExpression setex: - m = GetValueFromSet(setex); + m = GetValueFromSet(setex, lookupOptions); break; case TupleExpression tex: - m = GetValueFromTuple(tex); + m = GetValueFromTuple(tex, lookupOptions); break; case YieldExpression yex: - m = GetValueFromExpression(yex.Expression); + m = GetValueFromExpression(yex.Expression, lookupOptions); break; case GeneratorExpression genex: - m = GetValueFromGenerator(genex); + m = GetValueFromGenerator(genex, lookupOptions); break; case Comprehension comp: - m = GetValueFromComprehension(comp); + m = GetValueFromComprehension(comp, lookupOptions); break; case LambdaExpression lambda: m = GetValueFromLambda(lambda); @@ -182,14 +184,14 @@ public IMember GetValueFromExpression(Expression expr, LookupOptions options = L m = GetValueFromFormatSpecifier(formatSpecifier); break; case NamedExpression namedExpr: - m = GetValueFromExpression(namedExpr.Value); + m = GetValueFromExpression(namedExpr.Value, lookupOptions); break; // indexing with nothing, e.g Generic[] case ErrorExpression error: m = null; break; default: - m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr); + m = GetValueFromBinaryOp(expr, lookupOptions) ?? GetConstantFromLiteral(expr); break; } if (m == null) { @@ -232,19 +234,21 @@ private IMember GetValueFromName(NameExpression expr, LookupOptions options = Lo return UnknownType; } - private IMember GetValueFromMember(MemberExpression expr) { - if (expr?.Target == null || string.IsNullOrEmpty(expr.Name)) { + private IMember GetValueFromMember(MemberExpression expr, LookupOptions lookupOptions = LookupOptions.Normal) { + var memberName = expr?.Name; + if (expr?.Target == null || string.IsNullOrEmpty(memberName)) { return null; } - var m = GetValueFromExpression(expr.Target); + var m = GetValueFromExpression(expr.Target, lookupOptions); if (m == null) { return UnknownType; } var type = m.GetPythonType(); - var value = type?.GetMember(expr.Name); - type?.AddMemberReference(expr.Name, this, GetLocationOfName(expr)); + var value = type?.GetMember(memberName); + var location = GetLocationOfName(expr); + type?.AddMemberReference(memberName, this, location); if (type is IPythonModule) { return value; @@ -258,7 +262,7 @@ private IMember GetValueFromMember(MemberExpression expr) { f.AddReference(GetLocationOfName(expr)); return f.ToUnbound(); } - instance = type.CreateInstance(ArgumentSet.Empty(expr, this)); + instance = type.CreateInstance(ArgumentSet.Empty(expr, this)) as IPythonInstance; } instance = instance ?? m as IPythonInstance; @@ -286,13 +290,13 @@ private IMember GetValueFromMember(MemberExpression expr) { } } - private IMember GetValueFromConditional(ConditionalExpression expr) { + private IMember GetValueFromConditional(ConditionalExpression expr, LookupOptions lookupOptions = LookupOptions.Normal) { if (expr == null) { return null; } - var trueValue = GetValueFromExpression(expr.TrueExpression); - var falseValue = GetValueFromExpression(expr.FalseExpression); + var trueValue = GetValueFromExpression(expr.TrueExpression, lookupOptions); + var falseValue = GetValueFromExpression(expr.FalseExpression, lookupOptions); return trueValue ?? falseValue ?? UnknownType; } diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/FunctionCallEvaluator.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/FunctionCallEvaluator.cs index 4d0ccdda9..2790dee88 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Evaluation/FunctionCallEvaluator.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Evaluation/FunctionCallEvaluator.cs @@ -48,22 +48,13 @@ public IMember EvaluateCall(IArgumentSet args) { // Open scope and declare parameters using (_eval.OpenScope(_declaringModule, _function, out _)) { args.DeclareParametersInScope(_eval); - _function.Body.Walk(this); + _function.Body?.Walk(this); } return _result; } - public override bool Walk(AssignmentStatement node) { - foreach (var lhs in node.Left) { - if (lhs is NameExpression nameExp && (nameExp.Name == "self" || nameExp.Name == "cls")) { - return true; // Don't assign to 'self' or 'cls'. - } - } - return base.Walk(node); - } - public override bool Walk(ReturnStatement node) { - var value = Eval.GetValueFromExpression(node.Expression); + var value = Eval.GetValueFromExpression(node.Expression, LookupOptions.Normal); if (!value.IsUnknown()) { _result = value; return false; diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs index e6c948731..2ced129f3 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs @@ -13,29 +13,39 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System.Diagnostics; +using System.Linq; +using Microsoft.Python.Analysis.Specializations.Typing; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Values; -using Microsoft.Python.Core; using Microsoft.Python.Parsing.Ast; -using System.Diagnostics; -using System.Linq; namespace Microsoft.Python.Analysis.Analyzer.Handlers { internal sealed class AssignmentHandler : StatementHandler { public AssignmentHandler(AnalysisWalker walker) : base(walker) { } - public void HandleAssignment(AssignmentStatement node) { + public void HandleAssignment(AssignmentStatement node, LookupOptions lookupOptions = LookupOptions.Normal) { if (node.Right is ErrorExpression) { return; } // Filter out parenthesis expression in assignment because it makes no difference. - var lhs = node.Left.Select(s => s.RemoveParenthesis()); + var lhs = node.Left.Select(s => s.RemoveParenthesis()).ToArray(); - // TODO: Assigning like this is wrong; the assignment needs to be considering the - // right side's unpacking for what's on the left, not just apply it to every case. - var value = ExtractRhs(node.Right, lhs.FirstOrDefault()); + // Note that this is handling assignments of the same value to multiple variables, + // i.e. with "x = y = z = value", x/y/z are the items in lhs. If an expression looks + // like "x, y, z = value", then "x, y, z" is a *single* lhs value and its unpacking + // will be handled by AssignToExpr. + var value = ExtractRhs(node.Right, lhs.FirstOrDefault(), lookupOptions); if (value != null) { + // Named tuple may get assigned to variable that has name different from the tuple itself. + // Then the name may end up conflicting with other types in module when stub merges with + // module types. For example, 'tokenize' stub declares _TokenInfo = NamedTuple('TokenInfo', ...) + // but there is also 'class TokenInfo(_TokenInfo)' so we have to use the variable name + // in order to avoid type naming conflicts. + if (value is ITypingNamedTupleType nt && lhs.Length == 1 && lhs[0] is NameExpression nex) { + nt.SetName(nex.Name); + } foreach (var expr in lhs) { AssignToExpr(expr, value); } @@ -49,14 +59,15 @@ public void HandleNamedExpression(NamedExpression node) { var lhs = node.Target.RemoveParenthesis(); + // This is fine, as named expression targets are not allowed to be anything but simple names. var value = ExtractRhs(node.Value, lhs); if (value != null) { AssignToExpr(lhs, value); } } - private IMember ExtractRhs(Expression rhs, Expression typed) { - var value = Eval.GetValueFromExpression(rhs) ?? Eval.UnknownType; + private IMember ExtractRhs(Expression rhs, Expression typed, LookupOptions lookupOptions = LookupOptions.Normal) { + var value = Eval.GetValueFromExpression(rhs, lookupOptions) ?? Eval.UnknownType; // Check PEP hint first var valueType = Eval.GetTypeFromPepHint(rhs); @@ -86,7 +97,7 @@ private void AssignToExpr(Expression expr, IMember value) { HandleAnnotatedExpression(annExpr, value); break; case NameExpression nameExpr: - HandleNameExpression(nameExpr, value); + AssignVariable(nameExpr, value); break; case MemberExpression memberExpr: TryHandleClassVariable(memberExpr, value); @@ -94,35 +105,12 @@ private void AssignToExpr(Expression expr, IMember value) { } } - private bool IsValidAssignment(string name, Location loc) => !Eval.GetInScope(name).IsDeclaredAfter(loc); - - private void HandleNameExpression(NameExpression ne, IMember value) { - IScope scope; - if (Eval.CurrentScope.NonLocals[ne.Name] != null) { - Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Nonlocal); - scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne)); - return; - } - - if (Eval.CurrentScope.Globals[ne.Name] != null) { - Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Global); - scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne)); - return; - } - - var source = value.IsGeneric() ? VariableSource.Generic : VariableSource.Declaration; - var location = Eval.GetLocationOfName(ne); - if (IsValidAssignment(ne.Name, location)) { - Eval.DeclareVariable(ne.Name, value ?? Module.Interpreter.UnknownType, source, location); - } - } - - public void HandleAnnotatedExpression(ExpressionWithAnnotation expr, IMember value) { + public void HandleAnnotatedExpression(ExpressionWithAnnotation expr, IMember value, LookupOptions lookupOptions = LookupOptions.Normal) { if (expr?.Annotation == null) { return; } - var variableType = Eval.GetTypeFromAnnotation(expr.Annotation); + var variableType = Eval.GetTypeFromAnnotation(expr.Annotation, lookupOptions); // If value is null, then this is a pure declaration like // x: List[str] // without a value. If value is provided, then this is @@ -131,7 +119,7 @@ public void HandleAnnotatedExpression(ExpressionWithAnnotation expr, IMember val } private void TryHandleClassVariable(MemberExpression mex, IMember value) { - if (!string.IsNullOrEmpty(mex?.Name) && mex.Target is NameExpression nex && nex.Name.EqualsOrdinal("self")) { + if (mex.Target is NameExpression nex && nex.Name == "self") { var m = Eval.LookupNameInScopes(nex.Name, out _, LookupOptions.Local); var cls = m.GetPythonType(); if (cls != null) { diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs index 3374d04b2..3ea2f5411 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs @@ -14,7 +14,6 @@ // permissions and limitations under the License. using System.Collections.Generic; -using System.Diagnostics; using System.Linq; using Microsoft.Python.Analysis.Core.DependencyResolution; using Microsoft.Python.Analysis.Modules; @@ -41,79 +40,106 @@ public bool HandleFromImport(FromImportStatement node) { } var imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node); - if (HandleImportSearchResult(imports, null, null, node.Root, out var variableModule)) { - AssignVariables(node, imports, variableModule); - } + HandleImportSearchResult(imports, null, null, node.Root, out var variableModule); + AssignVariables(node, imports, variableModule); return false; } private void AssignVariables(FromImportStatement node, IImportSearchResult imports, PythonVariableModule variableModule) { - if (variableModule == null) { - return; - } - var names = node.Names; var asNames = node.AsNames; - if (names.Count == 1 && names[0].Name == "*") { + if (variableModule != null && names.Count == 1 && names[0].Name == "*") { // TODO: warn this is not a good style per // TODO: https://docs.python.org/3/faq/programming.html#what-are-the-best-practices-for-using-import-in-a-module // TODO: warn this is invalid if not in the global scope. - HandleModuleImportStar(variableModule, imports is ImplicitPackageImport, node.StartIndex); + HandleModuleImportStar(variableModule, imports, node.StartIndex, names[0]); return; } for (var i = 0; i < names.Count; i++) { var memberName = names[i].Name; - if (!string.IsNullOrEmpty(memberName)) { - var nameExpression = asNames[i] ?? names[i]; - var variableName = nameExpression?.Name ?? memberName; - if (!string.IsNullOrEmpty(variableName)) { - var variable = variableModule.Analysis?.GlobalScope?.Variables[memberName]; - var exported = variable ?? variableModule.GetMember(memberName); - var value = exported ?? GetValueFromImports(variableModule, imports as IImportChildrenSource, memberName); - // Do not allow imported variables to override local declarations - Eval.DeclareVariable(variableName, value, VariableSource.Import, nameExpression, CanOverwriteVariable(variableName, node.StartIndex)); - } + if (string.IsNullOrEmpty(memberName)) { + continue; + } + + var nameExpression = asNames[i] ?? names[i]; + var variableName = nameExpression?.Name ?? memberName; + if (!string.IsNullOrEmpty(variableName)) { + DeclareVariable(variableModule, memberName, imports, variableName, node.StartIndex, nameExpression); } } } - private void HandleModuleImportStar(PythonVariableModule variableModule, bool isImplicitPackage, int importPosition) { + private void HandleModuleImportStar(PythonVariableModule variableModule, IImportSearchResult imports, int importPosition, NameExpression nameExpression) { if (variableModule.Module == Module) { // from self import * won't define any new members return; } - // If __all__ is present, take it, otherwise declare all members from the module that do not begin with an underscore. - var memberNames = isImplicitPackage + var memberNames = imports is ImplicitPackageImport ? variableModule.GetMemberNames() - : variableModule.Analysis.StarImportMemberNames ?? variableModule.GetMemberNames().Where(s => !s.StartsWithOrdinal("_")); + : variableModule.Analysis.StarImportMemberNames + ?? variableModule.GetMemberNames().Where(s => !s.StartsWithOrdinal("_")).ToArray(); foreach (var memberName in memberNames) { - var member = variableModule.GetMember(memberName); - if (member == null) { - Log?.Log(TraceEventType.Verbose, $"Undefined import: {variableModule.Name}, {memberName}"); - } else if (member.MemberType == PythonMemberType.Unknown) { - Log?.Log(TraceEventType.Verbose, $"Unknown import: {variableModule.Name}, {memberName}"); - } - - member = member ?? Eval.UnknownType; - if (member is IPythonModule m) { - ModuleResolution.GetOrLoadModule(m.Name); - } + DeclareVariable(variableModule, memberName, imports, memberName, importPosition, nameExpression); + } + } + /// + /// Determines value of the variable and declares it. Value depends if source module has submodule + /// that is named the same as the variable and/or it has internal variables named same as the submodule. + /// + /// 'from a.b import c' when 'c' is both submodule of 'b' and a variable declared inside 'b'. + /// Source module of the variable such as 'a.b' in 'from a.b import c as d'. May be null if the module was not found. + /// Module member name such as 'c' in 'from a.b import c as d'. + /// Import search result. + /// Name of the variable to declare, such as 'd' in 'from a.b import c as d'. + /// Position of the import statement. + /// Location of the variable name expression. + private void DeclareVariable(PythonVariableModule variableModule, string memberName, IImportSearchResult imports, string variableName, int importPosition, Node nameLocation) { + IMember value = Eval.UnknownType; + + if (variableModule != null) { + // First try imports since child modules should win, i.e. in 'from a.b import c' + // 'c' should be a submodule if 'b' has one, even if 'b' also declares 'c = 1'. + value = GetValueFromImports(variableModule, imports as IImportChildrenSource, memberName); + + // First try exported or child submodules. + value = value ?? variableModule.GetMember(memberName); + + // Value may be variable or submodule. If it is variable, we need it in order to add reference. var variable = variableModule.Analysis?.GlobalScope?.Variables[memberName]; - // Do not allow imported variables to override local declarations - Eval.DeclareVariable(memberName, variable ?? member, VariableSource.Import, Eval.DefaultLocation, CanOverwriteVariable(memberName, importPosition)); + value = variable?.Value?.Equals(value) == true ? variable : value; + + // If nothing is exported, variables are still accessible. + value = value ?? variableModule.Analysis?.GlobalScope?.Variables[memberName]?.Value ?? Eval.UnknownType; + } + + // Do not allow imported variables to override local declarations + var canOverwrite = CanOverwriteVariable(variableName, importPosition, value); + + // Do not declare references to '*' + var locationExpression = nameLocation is NameExpression nex && nex.Name == "*" ? null : nameLocation; + Eval.DeclareVariable(variableName, value, VariableSource.Import, locationExpression, canOverwrite); + + // Make sure module is loaded and analyzed. + if (value is IPythonModule m) { + ModuleResolution.GetOrLoadModule(m.Name); } } - private bool CanOverwriteVariable(string name, int importPosition) { + private bool CanOverwriteVariable(string name, int importPosition, IMember newValue) { var v = Eval.CurrentScope.Variables[name]; if (v == null) { return true; // Variable does not exist } + + if (newValue.IsUnknown()) { + return false; // Do not overwrite potentially good value with unknowns. + } + // Allow overwrite if import is below the variable. Consider // x = 1 // x = 2 @@ -125,13 +151,14 @@ private bool CanOverwriteVariable(string name, int importPosition) { // is imported from another module. OK to overwrite. return true; } + var firstAssignmentPosition = references.Min(r => r.Span.ToIndexSpan(Ast).Start); return firstAssignmentPosition < importPosition; } private IMember GetValueFromImports(PythonVariableModule parentModule, IImportChildrenSource childrenSource, string memberName) { if (childrenSource == null || !childrenSource.TryGetChildImport(memberName, out var childImport)) { - return Interpreter.UnknownType; + return null; } switch (childImport) { @@ -141,7 +168,7 @@ private IMember GetValueFromImports(PythonVariableModule parentModule, IImportCh case ImplicitPackageImport packageImport: return GetOrCreateVariableModule(packageImport.FullName, parentModule, memberName); default: - return Interpreter.UnknownType; + return null; } } @@ -155,13 +182,13 @@ private void SpecializeFuture(FromImportStatement node) { var fn = new PythonFunctionType("print", new Location(Module), null, string.Empty); var o = new PythonFunctionOverload(fn, new Location(Module)); var parameters = new List { - new ParameterInfo("*values", Interpreter.GetBuiltinType(BuiltinTypeId.Object), ParameterKind.List, null), + new ParameterInfo("values", Interpreter.GetBuiltinType(BuiltinTypeId.Object), ParameterKind.List, null), new ParameterInfo("sep", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.KeywordOnly, null), new ParameterInfo("end", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.KeywordOnly, null), new ParameterInfo("file", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.KeywordOnly, null) }; o.SetParameters(parameters); - o.SetReturnValue(Interpreter.GetBuiltinType(BuiltinTypeId.NoneType), true); + o.SetReturnValue(Interpreter.GetBuiltinType(BuiltinTypeId.None), true); fn.AddOverload(o); Eval.DeclareVariable("print", fn, VariableSource.Import, printNameExpression); } diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs index b18b14883..4cdf23d8b 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs @@ -56,27 +56,47 @@ private void HandleImport(ModuleName moduleImportExpression, NameExpression asNa // import_module('fob.oar.baz') var importNames = ImmutableArray.Empty; var lastModule = default(PythonVariableModule); - var firstModule = default(PythonVariableModule); - foreach (var nameExpression in moduleImportExpression.Names) { + var resolvedModules = new (string name, PythonVariableModule module)[moduleImportExpression.Names.Count]; + for (var i = 0; i < moduleImportExpression.Names.Count; i++) { + var nameExpression = moduleImportExpression.Names[i]; importNames = importNames.Add(nameExpression.Name); var imports = ModuleResolution.CurrentPathResolver.GetImportsFromAbsoluteName(Module.FilePath, importNames, forceAbsolute); if (!HandleImportSearchResult(imports, lastModule, asNameExpression, moduleImportExpression, out lastModule)) { lastModule = default; break; } - - if (firstModule == default) { - firstModule = lastModule; - } + resolvedModules[i] = (nameExpression.Name, lastModule); } // "import fob.oar.baz as baz" is handled as baz = import_module('fob.oar.baz') - // "import fob.oar.baz" is handled as fob = import_module('fob') - if (!string.IsNullOrEmpty(asNameExpression?.Name) && lastModule != default) { + if (!string.IsNullOrEmpty(asNameExpression?.Name) && lastModule != null) { Eval.DeclareVariable(asNameExpression.Name, lastModule, VariableSource.Import, asNameExpression); - } else if (firstModule != default && !string.IsNullOrEmpty(importNames[0])) { - var firstName = moduleImportExpression.Names[0]; - Eval.DeclareVariable(importNames[0], firstModule, VariableSource.Import, firstName); + return; + } + + var firstModule = resolvedModules.Length > 0 ? resolvedModules[0].module : null; + var secondModule = resolvedModules.Length > 1 ? resolvedModules[1].module : null; + + // "import fob.oar.baz" when 'fob' is THIS module handled by declaring 'oar' as member. + // Consider pandas that has 'import pandas.testing' in __init__.py and 'testing' + // is available as member. See also https://github.com/microsoft/python-language-server/issues/1395 + if (firstModule?.Module == Eval.Module && importNames.Count > 1 && !string.IsNullOrEmpty(importNames[1]) && secondModule != null) { + Eval.DeclareVariable(importNames[0], firstModule, VariableSource.Import, moduleImportExpression.Names[0]); + Eval.DeclareVariable(importNames[1], secondModule, VariableSource.Import, moduleImportExpression.Names[1]); + } else { + // "import fob.oar.baz" is handled as fob = import_module('fob') + if (firstModule != null && !string.IsNullOrEmpty(importNames[0])) { + Eval.DeclareVariable(importNames[0], firstModule, VariableSource.Import, moduleImportExpression.Names[0]); + } + } + + // import a.b.c.d => declares a, b in the current module, c in b, d in c. + for (var i = 1; i < resolvedModules.Length - 1; i++) { + var (childName, childModule) = resolvedModules[i + 1]; + if (!string.IsNullOrEmpty(childName) && childModule != null) { + var parent = resolvedModules[i].module; + parent?.AddChildModule(childName, childModule); + } } } @@ -92,7 +112,7 @@ private bool HandleImportSearchResult(in IImportSearchResult imports, in PythonV return TryGetPackageFromImport(packageImport, parent, out variableModule); case RelativeImportBeyondTopLevel importBeyondTopLevel: var message = Resources.ErrorRelativeImportBeyondTopLevel.FormatInvariant(importBeyondTopLevel.RelativeImportName); - Eval.ReportDiagnostics(Eval.Module.Uri, + Eval.ReportDiagnostics(Eval.Module.Uri, new DiagnosticsEntry(message, location.GetLocation(Eval).Span, ErrorCodes.UnresolvedImport, Severity.Warning, DiagnosticSource.Analysis)); variableModule = default; return false; @@ -132,7 +152,7 @@ private bool TryGetModulePossibleImport(PossibleModuleImport possibleModuleImpor var fullName = possibleModuleImport.PrecedingModuleFullName; var module = ModuleResolution.GetOrLoadModule(possibleModuleImport.PrecedingModuleFullName); - if (module == default) { + if (module == null) { MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, fullName, location); return false; } @@ -157,7 +177,7 @@ private bool TryGetModulePossibleImport(PossibleModuleImport possibleModuleImpor return false; } } - + return true; } @@ -170,30 +190,28 @@ private void MakeUnresolvedImport(string variableName, string moduleName, Node l if (!string.IsNullOrEmpty(variableName)) { Eval.DeclareVariable(variableName, new SentinelModule(moduleName, Eval.Services), VariableSource.Import, location); } - Eval.ReportDiagnostics(Eval.Module.Uri, - new DiagnosticsEntry(Resources.ErrorUnresolvedImport.FormatInvariant(moduleName), + Eval.ReportDiagnostics(Eval.Module.Uri, + new DiagnosticsEntry(Resources.ErrorUnresolvedImport.FormatInvariant(moduleName), Eval.GetLocationInfo(location).Span, ErrorCodes.UnresolvedImport, Severity.Warning, DiagnosticSource.Analysis)); } private PythonVariableModule GetOrCreateVariableModule(in string fullName, in PythonVariableModule parentModule, in string memberName) { - if (_variableModules.TryGetValue(fullName, out var variableModule)) { - return variableModule; + if (!_variableModules.TryGetValue(fullName, out var variableModule)) { + variableModule = new PythonVariableModule(fullName, Eval.Interpreter); + _variableModules[fullName] = variableModule; } - variableModule = new PythonVariableModule(fullName, Eval.Interpreter); - _variableModules[fullName] = variableModule; parentModule?.AddChildModule(memberName, variableModule); return variableModule; } private PythonVariableModule GetOrCreateVariableModule(in IPythonModule module, in PythonVariableModule parentModule, in string memberName) { var moduleFullName = module.Name; - if (_variableModules.TryGetValue(moduleFullName, out var variableModule)) { - return variableModule; + if (!_variableModules.TryGetValue(moduleFullName, out var variableModule)) { + variableModule = new PythonVariableModule(module); + _variableModules[moduleFullName] = variableModule; } - variableModule = new PythonVariableModule(module); - _variableModules[moduleFullName] = variableModule; parentModule?.AddChildModule(memberName, variableModule); return variableModule; } diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/LoopHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/LoopHandler.cs index f568e31ac..c80045853 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Handlers/LoopHandler.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Handlers/LoopHandler.cs @@ -44,9 +44,11 @@ public bool HandleFor(ForStatement node) { return false; } - public void HandleWhile(WhileStatement node) { + public bool HandleWhile(WhileStatement node) { + node.Test?.Walk(Walker); node.Body?.Walk(Walker); node.ElseStatement?.Walk(Walker); + return false; } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/SequenceExpressionHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/SequenceExpressionHandler.cs index ac066f9da..4271a0549 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Handlers/SequenceExpressionHandler.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Handlers/SequenceExpressionHandler.cs @@ -13,9 +13,6 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. -using System.Collections.Generic; -using System.Linq; -using Microsoft.Python.Analysis.Analyzer.Evaluation; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Utilities; using Microsoft.Python.Analysis.Values; @@ -25,57 +22,39 @@ namespace Microsoft.Python.Analysis.Analyzer.Handlers { internal sealed class SequenceExpressionHandler : StatementHandler { public SequenceExpressionHandler(AnalysisWalker walker) : base(walker) { } - public void HandleAssignment(SequenceExpression seq, IMember value) { - Assign(seq, value, Eval); - } + public void HandleAssignment(SequenceExpression seq, IMember value) => Assign(seq, value); - internal static void Assign(SequenceExpression seq, IMember value, ExpressionEval eval) { - var typeEnum = new ValueEnumerator(value, eval.UnknownType); - Assign(seq, typeEnum, eval); + private void Assign(SequenceExpression seq, IMember value) { + var typeEnum = new ValueEnumerator(value, Eval.UnknownType, Eval.Module); + Assign(seq, typeEnum); } - private static void Assign(SequenceExpression seq, ValueEnumerator valueEnum, ExpressionEval eval) { + private void Assign(SequenceExpression seq, ValueEnumerator valueEnum) { foreach (var item in seq.Items) { switch (item) { case StarredExpression stx when stx.Expression is NameExpression nex && !string.IsNullOrEmpty(nex.Name): - eval.DeclareVariable(nex.Name, valueEnum.Next, VariableSource.Declaration, nex); + AssignVariable(nex, valueEnum.Next()); break; case ParenthesisExpression pex when pex.Expression is NameExpression nex && !string.IsNullOrEmpty(nex.Name): - eval.DeclareVariable(nex.Name, valueEnum.Next, VariableSource.Declaration, nex); + AssignVariable(nex, valueEnum.Next()); break; case NameExpression nex when !string.IsNullOrEmpty(nex.Name): - eval.DeclareVariable(nex.Name, valueEnum.Next, VariableSource.Declaration, nex); + AssignVariable(nex, valueEnum.Next()); break; // Nested sequence expression in sequence, Tuple[Tuple[int, str], int], List[Tuple[int], str] // TODO: Because of bug with how collection types are constructed, they don't make nested collection types // into instances, meaning we have to create it here case SequenceExpression se when valueEnum.Peek is IPythonCollection || valueEnum.Peek is IPythonCollectionType: - var collection = valueEnum.Next; + var collection = valueEnum.Next(); var pc = collection as IPythonCollection; var pct = collection as IPythonCollectionType; - Assign(se, pc ?? pct.CreateInstance(ArgumentSet.Empty(se, eval)), eval); + Assign(se, pc ?? pct.CreateInstance(ArgumentSet.Empty(se, Eval))); break; case SequenceExpression se: - Assign(se, valueEnum, eval); - break; - } - } - } - - private static IEnumerable NamesFromSequenceExpression(SequenceExpression rootSeq) { - var names = new List(); - foreach (var item in rootSeq.Items) { - var expr = item.RemoveParenthesis(); - switch (expr) { - case SequenceExpression seq: - names.AddRange(NamesFromSequenceExpression(seq)); - break; - case NameExpression nex: - names.Add(nex); + Assign(se, valueEnum); break; } } - return names; } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/StatementHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/StatementHandler.cs index 843fb41ed..81a6a4555 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Handlers/StatementHandler.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Handlers/StatementHandler.cs @@ -16,6 +16,7 @@ using Microsoft.Python.Analysis.Analyzer.Evaluation; using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core.Logging; using Microsoft.Python.Parsing.Ast; @@ -36,5 +37,27 @@ protected IModuleResolution ModuleResolution protected StatementHandler(AnalysisWalker walker) { Walker = walker; } + + protected void AssignVariable(NameExpression ne, IMember value) { + IScope scope; + if (Eval.CurrentScope.NonLocals[ne.Name] != null) { + Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Nonlocal); + scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne)); + return; + } + + if (Eval.CurrentScope.Globals[ne.Name] != null) { + Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Global); + scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne)); + return; + } + + var source = value.IsGeneric() ? VariableSource.Generic : VariableSource.Declaration; + var location = Eval.GetLocationOfName(ne); + if (IsValidAssignment(ne.Name, location)) { + Eval.DeclareVariable(ne.Name, value ?? Module.Interpreter.UnknownType, source, location); + } + } + private bool IsValidAssignment(string name, Location loc) => !Eval.GetInScope(name).IsDeclaredAfter(loc); } } diff --git a/src/Analysis/Ast/Impl/Analyzer/ProgressReporter.cs b/src/Analysis/Ast/Impl/Analyzer/ProgressReporter.cs index c5a87d1fc..47224a7f5 100644 --- a/src/Analysis/Ast/Impl/Analyzer/ProgressReporter.cs +++ b/src/Analysis/Ast/Impl/Analyzer/ProgressReporter.cs @@ -20,8 +20,8 @@ namespace Microsoft.Python.Analysis.Analyzer { internal sealed class ProgressReporter : IProgressReporter, IDisposable { - private const int _initialDelay = 50; - private const int _reportingInterval = 100; + private readonly static TimeSpan InitialDelay = TimeSpan.FromMilliseconds(50); + private readonly static TimeSpan ReportingInterval = TimeSpan.FromMilliseconds(100); private readonly IProgressService _progressService; private readonly object _lock = new object(); @@ -44,7 +44,7 @@ public void Dispose() { public void ReportRemaining(int count) { lock (_lock) { if (count == 0) { - EndReport(); + EndReport(); return; } @@ -52,7 +52,7 @@ public void ReportRemaining(int count) { // Delay reporting a bit in case the analysis is short in order to reduce UI flicker. _running = true; _reportTimer?.Dispose(); - _reportTimer = new Timer(OnReportTimer, null, _initialDelay, _reportingInterval); + _reportTimer = new Timer(OnReportTimer, null, InitialDelay, ReportingInterval); } _lastReportedCount = count; } diff --git a/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs b/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs index 3a1994ee6..c784d77b8 100644 --- a/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs +++ b/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs @@ -20,7 +20,6 @@ using System.Runtime.ExceptionServices; using System.Threading; using System.Threading.Tasks; -using Microsoft.Python.Analysis.Caching; using Microsoft.Python.Analysis.Dependencies; using Microsoft.Python.Analysis.Diagnostics; using Microsoft.Python.Analysis.Documents; @@ -36,7 +35,7 @@ namespace Microsoft.Python.Analysis.Analyzer { public sealed class PythonAnalyzer : IPythonAnalyzer, IDisposable { - private readonly IServiceManager _services; + private readonly IServiceContainer _services; private readonly IDependencyResolver _dependencyResolver; private readonly Dictionary _analysisEntries = new Dictionary(); private readonly DisposeToken _disposeToken = DisposeToken.Create(); @@ -51,7 +50,7 @@ public sealed class PythonAnalyzer : IPythonAnalyzer, IDisposable { private PythonAnalyzerSession _nextSession; private bool _forceGCOnNextSession; - public PythonAnalyzer(IServiceManager services, string cacheFolderPath = null) { + public PythonAnalyzer(IServiceContainer services) { _services = services; _log = services.GetService(); _dependencyResolver = new DependencyResolver(); @@ -59,9 +58,6 @@ public PythonAnalyzer(IServiceManager services, string cacheFolderPath = null) { _startNextSession = StartNextSession; _progress = new ProgressReporter(services.GetService()); - - _services.AddService(new CacheFolderService(_services, cacheFolderPath)); - _services.AddService(new StubCache(_services)); } public void Dispose() { @@ -69,19 +65,12 @@ public void Dispose() { _disposeToken.TryMarkDisposed(); } + #region IPythonAnalyzer public Task WaitForCompleteAnalysisAsync(CancellationToken cancellationToken = default) => _analysisCompleteEvent.WaitAsync(cancellationToken); public async Task GetAnalysisAsync(IPythonModule module, int waitTime, CancellationToken cancellationToken) { - PythonAnalyzerEntry entry; - lock (_syncObj) { - var key = new AnalysisModuleKey(module); - if (!_analysisEntries.TryGetValue(key, out entry)) { - var emptyAnalysis = new EmptyAnalysis(_services, (IDocument)module); - entry = new PythonAnalyzerEntry(emptyAnalysis); - _analysisEntries[key] = entry; - } - } + var entry = GetOrCreateAnalysisEntry(module, out _); if (waitTime < 0 || Debugger.IsAttached) { return await GetAnalysisAsync(entry, default, cancellationToken); @@ -114,13 +103,8 @@ private async Task GetAnalysisAsync(PythonAnalyzerEntry entry public void InvalidateAnalysis(IPythonModule module) { lock (_syncObj) { - var key = new AnalysisModuleKey(module); - if (_analysisEntries.TryGetValue(key, out var entry)) { - entry.Invalidate(_version + 1); - } else { - _analysisEntries[key] = new PythonAnalyzerEntry(new EmptyAnalysis(_services, (IDocument)module)); - _analysisCompleteEvent.Reset(); - } + var entry = GetOrCreateAnalysisEntry(module, out _); + entry.Invalidate(_version + 1); } } @@ -130,7 +114,6 @@ public void RemoveAnalysis(IPythonModule module) { key = new AnalysisModuleKey(module); _analysisEntries.Remove(key); } - _dependencyResolver.Remove(key); } @@ -142,11 +125,11 @@ public void EnqueueDocumentForAnalysis(IPythonModule module, ImmutableArray= bufferVersion) { - return; - } - - // It is possible that parsing request for the library has been started when document is open, - // but it is closed at the moment of analysis and then become open again. - // In this case, we still need to analyze the document, but using correct entry. - var nonUserAsDocumentKey = key.GetNonUserAsDocumentKey(); - if (entry.PreviousAnalysis is LibraryAnalysis && _analysisEntries.TryGetValue(nonUserAsDocumentKey, out var documentEntry)) { - key = nonUserAsDocumentKey; - entry = documentEntry; - } - - } else { - entry = new PythonAnalyzerEntry(new EmptyAnalysis(_services, (IDocument)module)); - _analysisEntries[key] = entry; - _analysisCompleteEvent.Reset(); + if (entry.BufferVersion >= bufferVersion) { + return; + } + // It is possible that parsing request for the library has been started when document is open, + // but it is closed at the moment of analysis and then become open again. + // In this case, we still need to analyze the document, but using correct entry. + var nonUserAsDocumentKey = key.GetNonUserAsDocumentKey(); + if (entry.PreviousAnalysis is LibraryAnalysis && _analysisEntries.TryGetValue(nonUserAsDocumentKey, out var documentEntry)) { + key = nonUserAsDocumentKey; + entry = documentEntry; } } - if (entry.Invalidate(module, ast, bufferVersion, version, out var dependencies)) { + var invalidate = entry.Invalidate(module, ast, bufferVersion, version, out var dependencies); + if (invalidate) { AnalyzeDocument(key, entry, dependencies); } } @@ -231,10 +207,44 @@ public IReadOnlyList LoadedModules { } public event EventHandler AnalysisComplete; + #endregion + + internal async Task RaiseAnalysisCompleteAsync(int moduleCount, double msElapsed) { + var notAllAnalyzed = false; + lock (_syncObj) { + if (_nextSession != null || _currentSession?.IsCompleted == false) { + return false; // There are active or pending sessions. + } + + var notAnalyzed = _analysisEntries.Values + .ExcludeDefault() + .Where(e => e.NotAnalyzed) + .ToArray(); + + notAllAnalyzed = notAnalyzed.Length > 0; + } + + if (notAllAnalyzed) { + // Attempt to see if within reasonable time new session starts + // This is a workaround since there may still be concurrency issues + // When module analysis session gets canceled and module never re-queued. + // We don't want to prevent event from firing when this [rarely] happens. + for (var i = 0; i < 20; i++) { + await Task.Delay(20); + lock (_syncObj) { + if (_analysisEntries.Values.ExcludeDefault().All(e => !e.NotAnalyzed)) { + break; // Now all modules are analyzed. + } + if (_nextSession != null || _currentSession?.IsCompleted == false) { + return false; // New sessions were created + } + } + } + } - internal void RaiseAnalysisComplete(int moduleCount, double msElapsed) { _analysisCompleteEvent.Set(); AnalysisComplete?.Invoke(this, new AnalysisCompleteEventArgs(moduleCount, msElapsed)); + return true; } private void AnalyzeDocument(in AnalysisModuleKey key, in PythonAnalyzerEntry entry, in ImmutableArray dependencies) { @@ -242,10 +252,10 @@ private void AnalyzeDocument(in AnalysisModuleKey key, in PythonAnalyzerEntry en ActivityTracker.StartTracking(); _log?.Log(TraceEventType.Verbose, $"Analysis of {entry.Module.Name} ({entry.Module.ModuleType}) queued. Dependencies: {string.Join(", ", dependencies.Select(d => d.IsTypeshed ? $"{d.Name} (stub)" : d.Name))}"); - var graphVersion = _dependencyResolver.ChangeValue(key, entry, entry.IsUserOrBuiltin || key.IsNonUserAsDocument, dependencies); + var graphVersion = _dependencyResolver.ChangeValue(key, entry, entry.IsUserOrBuiltin, dependencies); lock (_syncObj) { - if (_version > graphVersion) { + if (_version >= graphVersion) { return; } @@ -280,7 +290,7 @@ private bool TryCreateSession(in int graphVersion, in PythonAnalyzerEntry entry, return false; } - LoadMissingDocuments(entry.Module.Interpreter, walker.MissingKeys); + LoadMissingDocuments(entry.Module.Interpreter, walker); lock (_syncObj) { if (_currentSession == null) { @@ -329,41 +339,49 @@ private PythonAnalyzerSession CreateSession(in IDependencyChainWalker missingKeys) { - if (missingKeys.Count == 0) { - return; - } - - var foundKeys = ImmutableArray.Empty; - foreach (var missingKey in missingKeys) { + private void LoadMissingDocuments(IPythonInterpreter interpreter, IDependencyChainWalker walker) { + foreach (var key in walker.MissingKeys.ToArray()) { lock (_syncObj) { - if (_analysisEntries.TryGetValue(missingKey, out _)) { + if (_analysisEntries.TryGetValue(key, out _)) { + walker.MissingKeys = walker.MissingKeys.Remove(key); continue; } } - var (moduleName, _, isTypeshed) = missingKey; + var (moduleName, _, isTypeshed) = key; var moduleResolution = isTypeshed ? interpreter.TypeshedResolution : interpreter.ModuleResolution; + var module = moduleResolution.GetOrLoadModule(moduleName); if (module != null && module.ModuleType != ModuleType.Unresolved) { - foundKeys = foundKeys.Add(missingKey); + var entry = GetOrCreateAnalysisEntry(module, out _); + if (module.ModuleType == ModuleType.Specialized) { + walker.MissingKeys = walker.MissingKeys.Remove(key); + } else { + _dependencyResolver.TryAddValue(key, entry, entry.IsUserModule, ImmutableArray.Empty); + } } } + } - if (foundKeys.Count > 0) { - foreach (var foundKey in foundKeys) { - PythonAnalyzerEntry entry; - lock (_syncObj) { - if (!_analysisEntries.TryGetValue(foundKey, out entry)) { - continue; - } - } + private PythonAnalyzerEntry GetOrCreateAnalysisEntry(IPythonModule module, out AnalysisModuleKey key) { + key = new AnalysisModuleKey(module); + return GetOrCreateAnalysisEntry(module, key); + } - _dependencyResolver.TryAddValue(foundKey, entry, entry.IsUserModule, ImmutableArray.Empty); + private PythonAnalyzerEntry GetOrCreateAnalysisEntry(IPythonModule module, AnalysisModuleKey key) { + lock (_syncObj) { + if (_analysisEntries.TryGetValue(key, out var entry)) { + return entry; } + + var emptyAnalysis = new EmptyAnalysis(_services, (IDocument)module); + entry = new PythonAnalyzerEntry(emptyAnalysis); + _analysisEntries[key] = entry; + _analysisCompleteEvent.Reset(); + return entry; } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerEntry.cs b/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerEntry.cs index 02503c64c..16ffd9b90 100644 --- a/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerEntry.cs +++ b/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerEntry.cs @@ -21,6 +21,7 @@ using System.Threading.Tasks; using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Core; using Microsoft.Python.Core.Collections; using Microsoft.Python.Parsing.Ast; @@ -80,7 +81,7 @@ public int AnalysisVersion { } } - public bool NotAnalyzed => PreviousAnalysis is EmptyAnalysis; + public bool NotAnalyzed => PreviousAnalysis is EmptyAnalysis && Module.ModuleType != ModuleType.Specialized && Module.ModuleType != ModuleType.Builtins; public PythonAnalyzerEntry(EmptyAnalysis emptyAnalysis) { _previousAnalysis = emptyAnalysis; @@ -93,7 +94,7 @@ public PythonAnalyzerEntry(EmptyAnalysis emptyAnalysis) { } public Task GetAnalysisAsync(CancellationToken cancellationToken) - => _analysisTcs.Task.ContinueWith(t => t.GetAwaiter().GetResult(), cancellationToken); + => _analysisTcs.Task.WaitAsync(cancellationToken); public bool CanUpdateAnalysis(int version, out IPythonModule module, out PythonAst ast, out IDocumentAnalysis currentAnalysis) { lock (_syncObj) { @@ -271,7 +272,5 @@ private void UpdateAnalysisTcs(int analysisVersion) { _analysisTcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); } } - - } } diff --git a/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerSession.cs b/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerSession.cs index af049c36a..989a66c7b 100644 --- a/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerSession.cs +++ b/src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerSession.cs @@ -29,7 +29,6 @@ using Microsoft.Python.Analysis.Types; using Microsoft.Python.Core; using Microsoft.Python.Core.Logging; -using Microsoft.Python.Core.Services; using Microsoft.Python.Core.Testing; using Microsoft.Python.Parsing.Ast; @@ -42,11 +41,10 @@ internal sealed class PythonAnalyzerSession { private readonly PythonAnalyzerEntry _entry; private readonly Action _startNextSession; private readonly CancellationToken _analyzerCancellationToken; - private readonly IServiceManager _services; - private readonly AsyncManualResetEvent _analysisCompleteEvent; + private readonly IServiceContainer _services; private readonly IDiagnosticsService _diagnosticsService; private readonly IProgressReporter _progress; - private readonly IPythonAnalyzer _analyzer; + private readonly PythonAnalyzer _analyzer; private readonly ILogger _log; private readonly bool _forceGC; private readonly IModuleDatabaseService _moduleDatabaseService; @@ -66,9 +64,8 @@ public bool IsCompleted { public int Version { get; } public int AffectedEntriesCount { get; } - public PythonAnalyzerSession(IServiceManager services, + public PythonAnalyzerSession(IServiceContainer services, IProgressReporter progress, - AsyncManualResetEvent analysisCompleteEvent, Action startNextSession, CancellationToken analyzerCancellationToken, IDependencyChainWalker walker, @@ -77,7 +74,6 @@ public PythonAnalyzerSession(IServiceManager services, bool forceGC = false) { _services = services; - _analysisCompleteEvent = analysisCompleteEvent; _startNextSession = startNextSession; _analyzerCancellationToken = analyzerCancellationToken; Version = version; @@ -88,7 +84,7 @@ public PythonAnalyzerSession(IServiceManager services, _forceGC = forceGC; _diagnosticsService = _services.GetService(); - _analyzer = _services.GetService(); + _analyzer = _services.GetService(); _log = _services.GetService(); _moduleDatabaseService = _services.GetService(); _progress = progress; @@ -106,7 +102,7 @@ public void Start(bool analyzeEntry) { } if (analyzeEntry && _entry != null) { - Task.Run(() => AnalyzeEntry(), _analyzerCancellationToken).DoNotWait(); + Task.Run(AnalyzeEntry, _analyzerCancellationToken).DoNotWait(); } else { StartAsync().ContinueWith(_startNextSession, _analyzerCancellationToken).DoNotWait(); } @@ -144,7 +140,7 @@ private async Task StartAsync() { lock (_syncObj) { isCanceled = _isCanceled; _state = State.Completed; - isFinal = _walker.MissingKeys.Count == 0 && !isCanceled && remaining == 0; + isFinal = _walker.MissingKeys.Count == 0 && !_isCanceled && remaining == 0; _walker = null; } @@ -153,8 +149,9 @@ private async Task StartAsync() { if (isFinal) { var (modulesCount, totalMilliseconds) = ActivityTracker.EndTracking(); totalMilliseconds = Math.Round(totalMilliseconds, 2); - (_analyzer as PythonAnalyzer)?.RaiseAnalysisComplete(modulesCount, totalMilliseconds); - _log?.Log(TraceEventType.Verbose, $"Analysis complete: {modulesCount} modules in {totalMilliseconds} ms."); + if (await _analyzer.RaiseAnalysisCompleteAsync(modulesCount, totalMilliseconds)) { + _log?.Log(TraceEventType.Verbose, $"Analysis complete: {modulesCount} modules in {totalMilliseconds} ms."); + } } } } @@ -313,8 +310,8 @@ private bool CanUpdateAnalysis( // Library analysis exists, don't analyze again return false; } - if (ast == default) { - if (currentAnalysis == default) { + if (ast == null) { + if (currentAnalysis == null) { // Entry doesn't have ast yet. There should be at least one more session. Cancel(); _log?.Log(TraceEventType.Verbose, $"Analysis of {module.Name}({module.ModuleType}) canceled (no AST yet)."); @@ -356,10 +353,11 @@ private IDocumentAnalysis DoAnalyzeEntry(IDependencyChainNode node) { @@ -415,6 +413,11 @@ private IDocumentAnalysis CreateAnalysis(IDependencyChainNode(); + if (optionsProvider?.Options.KeepLibraryAst == true) { + createLibraryAnalysis = false; + } + if (!createLibraryAnalysis) { return new DocumentAnalysis(document, version, walker.GlobalScope, walker.Eval, walker.StarImportMemberNames); } diff --git a/src/Analysis/Ast/Impl/Analyzer/PythonInterpreter.cs b/src/Analysis/Ast/Impl/Analyzer/PythonInterpreter.cs index 556c7665c..832c081e6 100644 --- a/src/Analysis/Ast/Impl/Analyzer/PythonInterpreter.cs +++ b/src/Analysis/Ast/Impl/Analyzer/PythonInterpreter.cs @@ -137,20 +137,16 @@ public IPythonType GetBuiltinType(BuiltinTypeId id) { return type; } - if (id == BuiltinTypeId.NoneType) { - type = new PythonType("NoneType", new Location(_moduleResolution.BuiltinsModule), string.Empty, BuiltinTypeId.NoneType); - } else { - var bm = _moduleResolution.BuiltinsModule; - var typeName = id.GetTypeName(LanguageVersion); - if (typeName != null) { - type = _moduleResolution.BuiltinsModule.GetMember(typeName) as IPythonType; - } + var bm = _moduleResolution.BuiltinsModule; + var typeName = id.GetTypeName(LanguageVersion); + if (typeName != null) { + type = _moduleResolution.BuiltinsModule.GetMember(typeName) as IPythonType; + } + if (type == null) { + type = bm.GetAnyMember("__{0}__".FormatInvariant(id)) as IPythonType; if (type == null) { - type = bm.GetAnyMember("__{0}__".FormatInvariant(id)) as IPythonType; - if (type == null) { - return UnknownType; - } + return UnknownType; } } diff --git a/src/Analysis/Ast/Impl/Analyzer/StubMerger.cs b/src/Analysis/Ast/Impl/Analyzer/StubMerger.cs index d084e623d..188d41c0a 100644 --- a/src/Analysis/Ast/Impl/Analyzer/StubMerger.cs +++ b/src/Analysis/Ast/Impl/Analyzer/StubMerger.cs @@ -94,6 +94,36 @@ private void TransferTypesFromStub(IDocumentAnalysis stubAnalysis, CancellationT continue; } + var stubPrimaryModule = stubType.DeclaringModule.PrimaryModule; + + // If type comes from another module and stub type comes from that module stub, skip it. + // For example, 'sqlite3.dbapi2' has Date variable with value from 'datetime' module. + // Stub of 'sqlite3.dbapi2' also has Date from 'datetime (stub)'. We want to use + // type from the primary 'datetime' since it already merged its stub and updated + // type location and documentation while 'datetime' stub does not have documentation + // and its location is irrelevant since we don't navigate to stub source. + if (!_eval.Module.Equals(sourceType?.DeclaringModule) && + sourceType?.DeclaringModule.Stub != null && + sourceType.DeclaringModule.Equals(stubPrimaryModule)) { + continue; + } + + // If stub type is not from this module stub, redirect type to primary since primary has locations and documentation. + if (sourceType == null && stubPrimaryModule != null && !stubPrimaryModule.Equals(_eval.Module)) { + Debug.Assert(stubType.DeclaringModule.ModuleType == ModuleType.Stub); + switch (stubType) { + case PythonVariableModule vm: + stubType = vm.Module.PrimaryModule ?? stubType; + break; + case IPythonModule mod: + stubType = mod.PrimaryModule ?? stubType; + break; + default: + stubType = stubPrimaryModule.GetMember(v.Name)?.GetPythonType() ?? stubType; + break; + } + } + TryReplaceMember(v, sourceType, stubType, cancellationToken); } } @@ -110,8 +140,16 @@ private void TryReplaceMember(IVariable v, IPythonType sourceType, IPythonType s } break; - case PythonClassType sourceClass: - MergeClass(v, sourceClass, stubType, cancellationToken); + case IPythonClassType sourceClass: + MergeMembers(v, sourceClass, stubType, cancellationToken); + break; + + case PythonFunctionType sourceFunction: + MergeMembers(v, sourceFunction, stubType, cancellationToken); + break; + + case PythonPropertyType sourceProperty: + MergeMembers(v, sourceProperty, stubType, cancellationToken); break; case IPythonModule _: @@ -137,28 +175,38 @@ private void TryReplaceMember(IVariable v, IPythonType sourceType, IPythonType s } } - private void MergeClass(IVariable v, IPythonClassType sourceClass, IPythonType stubType, CancellationToken cancellationToken) { + private void MergeMembers(IVariable v, IPythonType sourceType, IPythonType stubType, CancellationToken cancellationToken) { // Transfer documentation first so we get class documentation // that comes from the class definition win over one that may // come from __init__ during the member merge below. - TransferDocumentationAndLocation(sourceClass, stubType); + TransferDocumentationAndLocation(sourceType.GetPythonType(), stubType); // Replace the class entirely since stub members may use generic types // and the class definition is important. We transfer missing members // from the original class to the stub. - _eval.DeclareVariable(v.Name, v.Value, v.Source); + // + // In case module is compiled, it is already a stub and has no locations + // for code navigation.. In this case we replace the entire variable by one + // from the stub rather than just the value since stub variable has location + // and its own root definition/reference chain. + if (sourceType.DeclaringModule.ModuleType == ModuleType.Compiled || + sourceType.DeclaringModule.ModuleType == ModuleType.CompiledBuiltin) { + _eval.ReplaceVariable(v); + } else { + _eval.DeclareVariable(v.Name, v.Value, v.Source); + } // First pass: go through source class members and pick those // that are not present in the stub class. - foreach (var name in sourceClass.GetMemberNames().ToArray()) { + foreach (var name in sourceType.GetMemberNames().ToArray()) { cancellationToken.ThrowIfCancellationRequested(); - var sourceMember = sourceClass.GetMember(name); + var sourceMember = sourceType.GetMember(name); if (sourceMember.IsUnknown()) { continue; // Do not add unknowns to the stub. } var sourceMemberType = sourceMember?.GetPythonType(); - if (sourceMemberType is IPythonClassMember cm && cm.DeclaringType != sourceClass) { + if (sourceMemberType is IPythonClassMember cm && !cm.DeclaringModule.Equals(sourceType.DeclaringModule)) { continue; // Only take members from this class and not from bases. } if (!IsFromThisModuleOrSubmodules(sourceMemberType)) { @@ -177,7 +225,7 @@ private void MergeClass(IVariable v, IPythonClassType sourceClass, IPythonType s continue; // Stub already have the member, don't replace. } - (stubType as PythonType)?.AddMember(name, stubMember, overwrite: true); + (stubType as PythonType)?.AddMember(name, sourceMember, overwrite: true); } // Second pass: go through stub class members and if they don't have documentation @@ -196,7 +244,7 @@ private void MergeClass(IVariable v, IPythonClassType sourceClass, IPythonType s continue; // Only take members from this class and not from bases. } - var sourceMember = sourceClass.GetMember(name); + var sourceMember = sourceType.GetMember(name); if (sourceMember.IsUnknown()) { continue; } @@ -299,10 +347,19 @@ private void TransferDocumentationAndLocation(IPythonType sourceType, IPythonTyp /// or location of unrelated types such as coming from the base object type. /// private bool IsFromThisModuleOrSubmodules(IPythonType type) { + if(type.IsUnknown()) { + return false; + } var thisModule = _eval.Module; var typeModule = type.DeclaringModule; var typeMainModuleName = typeModule.Name.Split('.').FirstOrDefault(); - return typeModule.Equals(thisModule) || typeMainModuleName == thisModule.Name; + if (typeModule.Equals(thisModule) || typeMainModuleName == thisModule.Name) { + return true; + } + // Check if module is explicitly imported by the current one. For example, 'os' + // imports 'nt' and os.pyi specifies functions from 'nt' such as mkdir and so on. + var imported = thisModule.GlobalScope.Variables[typeModule.Name]; + return imported?.Value != null && imported.Source == VariableSource.Import; } } } diff --git a/src/Analysis/Ast/Impl/Analyzer/Symbols/ClassEvaluator.cs b/src/Analysis/Ast/Impl/Analyzer/Symbols/ClassEvaluator.cs index c563428c5..cfc515212 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Symbols/ClassEvaluator.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Symbols/ClassEvaluator.cs @@ -90,10 +90,10 @@ private void ProcessClassBody() { foreach (var s in GetStatements(_classDef)) { switch (s) { case AssignmentStatement assignment: - AssignmentHandler.HandleAssignment(assignment); + AssignmentHandler.HandleAssignment(assignment, LookupOptions.All); break; case ExpressionStatement e: - AssignmentHandler.HandleAnnotatedExpression(e.Expression as ExpressionWithAnnotation, null); + AssignmentHandler.HandleAnnotatedExpression(e.Expression as ExpressionWithAnnotation, null, LookupOptions.All); break; } } @@ -113,7 +113,7 @@ private IEnumerable ProcessBases() { using (Eval.OpenScope(Eval.CurrentScope.OuterScope)) { var bases = new List(); foreach (var a in _classDef.Bases.Where(a => string.IsNullOrEmpty(a.Name))) { - if (IsValidBase(a)) { + if (IsValidBase(a, LookupOptions.Normal)) { TryAddBase(bases, a); } else { ReportInvalidBase(a); @@ -123,22 +123,15 @@ private IEnumerable ProcessBases() { } } - private bool IsValidBase(Arg a) { + private bool IsValidBase(Arg a, LookupOptions lookupOptions) { var expr = a.Expression; - var m = Eval.GetValueFromExpression(expr); + var m = Eval.GetValueFromExpression(expr, lookupOptions); // Allow any unknown members if (m.IsUnknown()) { return true; } - // Allow extensions from specialized functions - // We specialized type to be a function even though it is a class, so this allows extension of type - // TODO handle module specialization better: https://github.com/microsoft/python-language-server/issues/1367 - if (m is IPythonType t && t.IsSpecialized) { - return true; - } - switch (m.MemberType) { // Inheriting from these members is invalid case PythonMemberType.Method: diff --git a/src/Analysis/Ast/Impl/Analyzer/Symbols/FunctionEvaluator.cs b/src/Analysis/Ast/Impl/Analyzer/Symbols/FunctionEvaluator.cs index fb87bb274..5151a1a68 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Symbols/FunctionEvaluator.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Symbols/FunctionEvaluator.cs @@ -38,10 +38,9 @@ internal sealed class FunctionEvaluator : MemberEvaluator { public FunctionEvaluator(ExpressionEval eval, PythonFunctionOverload overload) : base(eval, overload.FunctionDefinition) { _overload = overload; - _function = overload.ClassMember ?? throw new NullReferenceException(nameof(overload.ClassMember)); + _function = overload.ClassMember ?? throw new ArgumentNullException(nameof(overload.ClassMember)); _self = _function.DeclaringType as PythonClassType; - - FunctionDefinition = overload.FunctionDefinition; + FunctionDefinition = overload.FunctionDefinition ?? throw new ArgumentNullException(nameof(overload.FunctionDefinition)); } private FunctionDefinition FunctionDefinition { get; } @@ -61,49 +60,64 @@ public override void Evaluate() { // Do process body of constructors since they may be declaring // variables that are later used to determine return type of other // methods and properties. + var optionsProvider = Eval.Services.GetService(); + var keepAst = optionsProvider?.Options.KeepLibraryAst == true; var ctor = _function.IsDunderInit() || _function.IsDunderNew(); - if (ctor || returnType.IsUnknown() || Module.ModuleType == ModuleType.User) { + if (ctor || returnType.IsUnknown() || Module.ModuleType == ModuleType.User || keepAst) { // Return type from the annotation is sufficient for libraries and stubs, no need to walk the body. FunctionDefinition.Body?.Walk(this); // For libraries remove declared local function variables to free up some memory // unless function has inner classes or functions. - var optionsProvider = Eval.Services.GetService(); - if (Module.ModuleType != ModuleType.User && - optionsProvider?.Options.KeepLibraryLocalVariables != true && + if (Module.ModuleType != ModuleType.User && !keepAst && Eval.CurrentScope.Variables.All( v => v.GetPythonType() == null && v.GetPythonType() == null) ) { - ((VariableCollection)Eval.CurrentScope.Variables).Clear(); + ((VariableCollection)Eval.CurrentScope.Variables).Clear(); } } } Result = _function; } - private IPythonType TryDetermineReturnValue() { - var annotationType = Eval.GetTypeFromAnnotation(FunctionDefinition.ReturnAnnotation); - if (!annotationType.IsUnknown()) { - // Annotations are typically types while actually functions return - // instances unless specifically annotated to a type such as Type[T]. - // TODO: try constructing argument set from types. Consider Tuple[_T1, _T2] where _T1 = TypeVar('_T1', str, bytes) - var t = annotationType.CreateInstance(ArgumentSet.Empty(FunctionDefinition.ReturnAnnotation, Eval)); - // If instance could not be created, such as when return type is List[T] and - // type of T is not yet known, just use the type. - var instance = t.IsUnknown() ? (IMember) annotationType : t; - _overload.SetReturnValue(instance, true); _overload.SetReturnValue(instance, true); - } else { - // Check if function is a generator - var suite = FunctionDefinition.Body as SuiteStatement; - var yieldExpr = suite?.Statements.OfType().Select(s => s.Expression as YieldExpression).ExcludeDefault().FirstOrDefault(); - if (yieldExpr != null) { - // Function return is an iterator - var yieldValue = Eval.GetValueFromExpression(yieldExpr.Expression) ?? Eval.UnknownType; - var returnValue = new PythonGenerator(Eval.Interpreter, yieldValue); - _overload.SetReturnValue(returnValue, true); - } + public static IMember GetReturnValueFromAnnotation(ExpressionEval eval, Expression annotation) { + if (eval == null || annotation == null) { + return null; } - return annotationType; + + var annotationType = eval.GetTypeFromAnnotation(annotation, LookupOptions.All); + if (annotationType.IsUnknown()) { + return null; + } + + // Annotations are typically types while actually functions return + // instances unless specifically annotated to a type such as Type[T]. + // TODO: try constructing argument set from types. Consider Tuple[_T1, _T2] where _T1 = TypeVar('_T1', str, bytes) + var t = annotationType.CreateInstance(ArgumentSet.Empty(annotation, eval)); + // If instance could not be created, such as when return type is List[T] and + // type of T is not yet known, just use the type. + var instance = t.IsUnknown() ? (IMember)annotationType : t; + return instance; + } + private IMember TryDetermineReturnValue() { + var returnType = GetReturnValueFromAnnotation(Eval, FunctionDefinition.ReturnAnnotation); + if (returnType != null) { + _overload.SetReturnValue(returnType, true); + return returnType; + } + + // Check if function is a generator + var suite = FunctionDefinition.Body as SuiteStatement; + var yieldExpr = suite?.Statements.OfType().Select(s => s.Expression as YieldExpression).ExcludeDefault().FirstOrDefault(); + if (yieldExpr != null) { + // Function return is an iterator + var yieldValue = Eval.GetValueFromExpression(yieldExpr.Expression) ?? Eval.UnknownType; + var returnValue = new PythonGenerator(Eval.Interpreter, yieldValue); + _overload.SetReturnValue(returnValue, true); + return returnValue; + } + + return null; } private void CheckValidOverload(IReadOnlyList parameters) { @@ -112,7 +126,7 @@ private void CheckValidOverload(IReadOnlyList parameters) { case IPythonFunctionType function: CheckValidFunction(function, parameters); break; - //TODO check properties + //TODO check properties } } } @@ -128,6 +142,16 @@ private void CheckValidFunction(IPythonFunctionType function, IReadOnlyList(memberExp.Name, value) }, false); - } - continue; + case MemberExpression memberExp when memberExp.Target is NameExpression nameExp1: + if (_function.DeclaringType.GetPythonType() is PythonClassType t && nameExp1.Name == "self") { + t.AddMembers(new[] { new KeyValuePair(memberExp.Name, value) }, false); } + continue; case NameExpression nameExp2 when nameExp2.Name == "self": - return true; // Don't assign to 'self' + // Only assign to 'self' if it is not declared yet. + if (Eval.LookupNameInScopes(nameExp2.Name, out _) == null) { + Eval.DeclareVariable(nameExp2.Name, value, VariableSource.Declaration); + } + return true; } } return base.Walk(node); @@ -181,7 +208,7 @@ public override bool Walk(ReturnStatement node) { if (value != null) { // although technically legal, __init__ in a constructor should not have a not-none return value - if (_function.IsDunderInit() && !value.IsOfType(BuiltinTypeId.NoneType)) { + if (_function.IsDunderInit() && !value.IsOfType(BuiltinTypeId.None)) { Eval.ReportDiagnostics(Module.Uri, new DiagnosticsEntry( Resources.ReturnInInit, node.GetLocation(Eval).Span, diff --git a/src/Analysis/Ast/Impl/Analyzer/Symbols/SymbolCollector.cs b/src/Analysis/Ast/Impl/Analyzer/Symbols/SymbolCollector.cs index b08c0f07e..07c1afb18 100644 --- a/src/Analysis/Ast/Impl/Analyzer/Symbols/SymbolCollector.cs +++ b/src/Analysis/Ast/Impl/Analyzer/Symbols/SymbolCollector.cs @@ -51,12 +51,14 @@ public override bool Walk(IfStatement node) => node.WalkIfWithSystemConditions(this, _eval.Ast.LanguageVersion, _eval.Services.GetService()); public override bool Walk(ClassDefinition cd) { - if (IsDeprecated(cd)) { - return false; - } - if (!string.IsNullOrEmpty(cd.NameExpression?.Name)) { var classInfo = CreateClass(cd); + if (classInfo == null) { + // we can't create class info for this node. + // don't walk down + return false; + } + // The variable is transient (non-user declared) hence it does not have location. // Class type is tracking locations for references and renaming. _eval.DeclareVariable(cd.Name, classInfo, VariableSource.Declaration); @@ -68,16 +70,14 @@ public override bool Walk(ClassDefinition cd) { } public override void PostWalk(ClassDefinition cd) { - if (!IsDeprecated(cd) && !string.IsNullOrEmpty(cd.NameExpression?.Name)) { + if (!string.IsNullOrEmpty(cd.NameExpression?.Name) && + _typeMap.ContainsKey(cd)) { _scopes.Pop().Dispose(); } base.PostWalk(cd); } public override bool Walk(FunctionDefinition fd) { - if (IsDeprecated(fd)) { - return false; - } if (!string.IsNullOrEmpty(fd.Name)) { AddFunctionOrProperty(fd); // Open function scope @@ -87,7 +87,7 @@ public override bool Walk(FunctionDefinition fd) { } public override void PostWalk(FunctionDefinition fd) { - if (!IsDeprecated(fd) && !string.IsNullOrEmpty(fd.Name)) { + if (!string.IsNullOrEmpty(fd.Name)) { _scopes.Pop().Dispose(); } base.PostWalk(fd); @@ -95,9 +95,14 @@ public override void PostWalk(FunctionDefinition fd) { private PythonClassType CreateClass(ClassDefinition cd) { PythonType declaringType = null; - if(!(cd.Parent is PythonAst)) { - Debug.Assert(_typeMap.ContainsKey(cd.Parent)); - _typeMap.TryGetValue(cd.Parent, out declaringType); + if (!(cd.Parent is PythonAst)) { + if (!_typeMap.TryGetValue(cd.Parent, out declaringType)) { + // we can get into this situation if parent is defined twice and we preserve + // only one of them. + // for example, code has function definition with exact same signature + // and class is defined under one of that function + return null; + } } var cls = new PythonClassType(cd, declaringType, _eval.GetLocationOfName(cd), _eval.SuppressBuiltinLookup ? BuiltinTypeId.Unknown : BuiltinTypeId.Type); @@ -120,6 +125,10 @@ private void AddFunction(FunctionDefinition fd, PythonType declaringType) { f = new PythonFunctionType(fd, declaringType, _eval.GetLocationOfName(fd)); // The variable is transient (non-user declared) hence it does not have location. // Function type is tracking locations for references and renaming. + + // if there are multiple functions with same name exist, only the very first one will be + // maintained in the scope. we should improve this if possible. + // https://github.com/microsoft/python-language-server/issues/1693 _eval.DeclareVariable(fd.Name, f, VariableSource.Declaration); _typeMap[fd] = f; declaringType?.AddMember(f.Name, f, overwrite: true); @@ -157,7 +166,7 @@ private PythonFunctionOverload GetOverloadFromStub(FunctionDefinition node) { if (t is IPythonFunctionType f) { return f.Overloads .OfType() - .FirstOrDefault(o => o.Parameters.Count == node.Parameters.Length); + .FirstOrDefault(o => o.Parameters.Count == node.Parameters.Where(p => !p.IsPositionalOnlyMarker).Count()); } return null; } @@ -223,18 +232,5 @@ private IMember GetMemberFromStub(string name) { return member; } - - private static bool IsDeprecated(ClassDefinition cd) - => cd.Decorators?.Decorators != null && IsDeprecated(cd.Decorators.Decorators); - - private static bool IsDeprecated(FunctionDefinition fd) - => fd.Decorators?.Decorators != null && IsDeprecated(fd.Decorators.Decorators); - - private static bool IsDeprecated(IEnumerable decorators) - => decorators.OfType().Any(IsDeprecationDecorator); - - private static bool IsDeprecationDecorator(CallExpression c) - => (c.Target is MemberExpression n1 && n1.Name == "deprecated") || - (c.Target is NameExpression n2 && n2.Name == "deprecated"); } } diff --git a/src/Analysis/Ast/Impl/Caching/CacheFolders.cs b/src/Analysis/Ast/Impl/Caching/CacheFolders.cs index 414dec1e3..3bd0b2e4a 100644 --- a/src/Analysis/Ast/Impl/Caching/CacheFolders.cs +++ b/src/Analysis/Ast/Impl/Caching/CacheFolders.cs @@ -16,28 +16,19 @@ using System; using System.Diagnostics; using System.IO; -using System.Security.Cryptography; -using System.Text; using Microsoft.Python.Core; using Microsoft.Python.Core.Logging; using Microsoft.Python.Core.OS; namespace Microsoft.Python.Analysis.Caching { - internal sealed class CacheFolderService: ICacheFolderService { + internal sealed class CacheFolderService : ICacheFolderService { public CacheFolderService(IServiceContainer services, string cacheRootFolder) { CacheFolder = cacheRootFolder ?? GetCacheFolder(services); } public string CacheFolder { get; } - public string GetFileNameFromContent(string content) { - // File name depends on the content so we can distinguish between different versions. - using (var hash = SHA256.Create()) { - return Convert - .ToBase64String(hash.ComputeHash(new UTF8Encoding(false).GetBytes(content))) - .Replace('/', '_').Replace('+', '-'); - } - } + public string GetFileNameFromContent(string content) => content.GetHashString(); private static string GetCacheFolder(IServiceContainer services) { var platform = services.GetService(); @@ -47,18 +38,18 @@ private static string GetCacheFolder(IServiceContainer services) { var localAppData = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData); var plsSubfolder = $"Microsoft{Path.DirectorySeparatorChar}Python Language Server"; var defaultCachePath = Path.Combine(localAppData, plsSubfolder); - + string cachePath = null; try { const string homeVarName = "HOME"; var homeFolderPath = Environment.GetEnvironmentVariable(homeVarName); - if(platform.IsWindows) { + if (platform.IsWindows) { cachePath = defaultCachePath; } if (platform.IsMac) { - if (CheckVariableSet(homeVarName, homeFolderPath, logger) + if (CheckVariableSet(homeVarName, homeFolderPath, logger) && CheckPathRooted(homeVarName, homeFolderPath, logger) && !string.IsNullOrWhiteSpace(homeFolderPath)) { cachePath = Path.Combine(homeFolderPath, "Library/Caches", plsSubfolder); diff --git a/src/Analysis/Ast/Impl/Caching/CacheService.cs b/src/Analysis/Ast/Impl/Caching/CacheService.cs new file mode 100644 index 000000000..664f8f9c7 --- /dev/null +++ b/src/Analysis/Ast/Impl/Caching/CacheService.cs @@ -0,0 +1,46 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Diagnostics; +using Microsoft.Python.Core; +using Microsoft.Python.Core.IO; +using Microsoft.Python.Core.Logging; +using Microsoft.Python.Core.Services; + +namespace Microsoft.Python.Analysis.Caching { + /// + /// Register caching services for the given cache folder path + /// + public static class CacheService { + public static void Register(IServiceManager services, string cacheFolderPath, bool pathCheck = true) { + var log = services.GetService(); + var fs = services.GetService(); + + // this is not thread safe. this is not supposed to be called concurrently + var cachingService = services.GetService(); + if (cachingService != null) { + return; + } + + if (pathCheck && cacheFolderPath != null && !fs.DirectoryExists(cacheFolderPath)) { + log?.Log(TraceEventType.Warning, Resources.Invalid_0_CacheFolder.FormatUI(cacheFolderPath)); + cacheFolderPath = null; + } + + services.AddService(new CacheFolderService(services, cacheFolderPath)); + services.AddService(new StubCache(services)); + } + } +} diff --git a/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseCache.cs b/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseCache.cs new file mode 100644 index 000000000..274259d70 --- /dev/null +++ b/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseCache.cs @@ -0,0 +1,37 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. +// using System; + +namespace Microsoft.Python.Analysis.Caching { + /// + /// Provides location of the analysis database cache. + /// + public interface IModuleDatabaseCache { + /// + /// Cache folder base name without version, such as 'analysis.v'. + /// + string CacheFolderBaseName { get; } + + /// + /// Database format version. + /// + int DatabaseFormatVersion { get; } + + /// + /// Full path to the cache folder includding version. + /// + string CacheFolder { get; } + } +} diff --git a/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseService.cs b/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseService.cs index 9477a952f..fb40a5bfc 100644 --- a/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseService.cs +++ b/src/Analysis/Ast/Impl/Caching/Definitions/IModuleDatabaseService.cs @@ -17,20 +17,9 @@ using System.Threading.Tasks; using Microsoft.Python.Analysis.Dependencies; using Microsoft.Python.Analysis.Types; -using Microsoft.Python.Analysis.Values; namespace Microsoft.Python.Analysis.Caching { - /// - /// Represents global scope that has been restored from - /// the database but has not been fully populated yet. - /// Used to attach to analysis so variables can be - /// accessed during classes and methods restoration. - /// - internal interface IRestoredGlobalScope : IGlobalScope { - void ReconstructVariables(); - } - - internal interface IModuleDatabaseService { + internal interface IModuleDatabaseService: IModuleDatabaseCache { /// /// Creates global scope from module persistent state. /// Global scope is then can be used to construct module analysis. @@ -55,10 +44,5 @@ internal interface IModuleDatabaseService { /// Determines if module analysis exists in the storage. /// bool ModuleExistsInStorage(string moduleName, string filePath); - - /// - /// Clear cached data. - /// - void Clear(); } } diff --git a/src/Analysis/Ast/Impl/Caching/Definitions/IRestoredGlobalScope.cs b/src/Analysis/Ast/Impl/Caching/Definitions/IRestoredGlobalScope.cs new file mode 100644 index 000000000..71bf8e130 --- /dev/null +++ b/src/Analysis/Ast/Impl/Caching/Definitions/IRestoredGlobalScope.cs @@ -0,0 +1,28 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using Microsoft.Python.Analysis.Values; + +namespace Microsoft.Python.Analysis.Caching { + /// + /// Represents global scope that has been restored from + /// the database but has not been fully populated yet. + /// Used to attach to analysis so variables can be + /// accessed during classes and methods restoration. + /// + internal interface IRestoredGlobalScope : IGlobalScope { + void ReconstructVariables(); + } +} diff --git a/src/Analysis/Ast/Impl/Caching/StubCache.cs b/src/Analysis/Ast/Impl/Caching/StubCache.cs index 58081d146..fcee35f12 100644 --- a/src/Analysis/Ast/Impl/Caching/StubCache.cs +++ b/src/Analysis/Ast/Impl/Caching/StubCache.cs @@ -23,7 +23,7 @@ namespace Microsoft.Python.Analysis.Caching { internal sealed class StubCache : IStubCache { - private const int _stubCacheFormatVersion = 2; + private const int _stubCacheFormatVersion = 4; private readonly IFileSystem _fs; private readonly ILogger _log; @@ -110,7 +110,7 @@ public void WriteCachedModule(string filePath, string code) { if (!string.IsNullOrEmpty(cache)) { _log?.Log(TraceEventType.Verbose, "Writing cached module: ", cache); // Don't block analysis on cache writes. - CacheWritingTask = Task.Run(() => _fs.WriteTextWithRetry(cache, code)); + CacheWritingTask = Task.Run(() => _fs.WriteTextWithRetry(cache, code, _log)); CacheWritingTask.DoNotWait(); } } diff --git a/src/Analysis/Ast/Impl/Definitions/AnalysisOptions.cs b/src/Analysis/Ast/Impl/Definitions/AnalysisOptions.cs index dcf73ec35..52a8f2106 100644 --- a/src/Analysis/Ast/Impl/Definitions/AnalysisOptions.cs +++ b/src/Analysis/Ast/Impl/Definitions/AnalysisOptions.cs @@ -36,13 +36,6 @@ public enum AnalysisCachingLevel { public class AnalysisOptions { public bool LintingEnabled { get; set; } - /// - /// Keep in memory information on local variables declared in - /// functions in libraries. Provides ability to navigate to - /// symbols used in function bodies in packages and libraries. - /// - public bool KeepLibraryLocalVariables { get; set; } - /// /// Keep in memory AST of library source code. May somewhat /// improve performance when library code has to be re-analyzed. diff --git a/src/Analysis/Ast/Impl/Dependencies/DependencyCollector.cs b/src/Analysis/Ast/Impl/Dependencies/DependencyCollector.cs index 586e314c1..a0c3e4cca 100644 --- a/src/Analysis/Ast/Impl/Dependencies/DependencyCollector.cs +++ b/src/Analysis/Ast/Impl/Dependencies/DependencyCollector.cs @@ -31,7 +31,7 @@ internal sealed class DependencyCollector { public DependencyCollector(IPythonModule module, bool? isTypeShed = null) { _module = module; - _isTypeshed = isTypeShed ?? module is StubPythonModule stub && stub.IsTypeshed; + _isTypeshed = isTypeShed ?? module.IsTypeshed; _moduleResolution = module.Interpreter.ModuleResolution; _pathResolver = _isTypeshed ? module.Interpreter.TypeshedResolution.CurrentPathResolver diff --git a/src/Analysis/Ast/Impl/Dependencies/DependencyResolver.cs b/src/Analysis/Ast/Impl/Dependencies/DependencyResolver.cs index 26c8e07ca..0813a6303 100644 --- a/src/Analysis/Ast/Impl/Dependencies/DependencyResolver.cs +++ b/src/Analysis/Ast/Impl/Dependencies/DependencyResolver.cs @@ -23,6 +23,9 @@ namespace Microsoft.Python.Analysis.Dependencies { internal sealed class DependencyResolver : IDependencyResolver { + // optimization to only analyze one that is reachable from root + private readonly bool _checkVertexReachability = true; + private readonly Dictionary _keys = new Dictionary(); private readonly List> _vertices = new List>(); private readonly object _syncObj = new object(); @@ -30,7 +33,7 @@ internal sealed class DependencyResolver : IDependencyResolver _version; - + public int ChangeValue(in TKey key, in TValue value, in bool isRoot, params TKey[] incomingKeys) => ChangeValue(key, value, isRoot, ImmutableArray.Create(incomingKeys)); @@ -53,7 +56,7 @@ public int TryAddValue(in TKey key, in TValue value, in bool isRoot, in Immutabl index = _keys.Count; _keys[key] = index; _vertices.Add(default); - } else if (_vertices[index] != default) { + } else if (_vertices[index] != null) { return _version; } @@ -71,14 +74,14 @@ public int Remove(in TKey key) { var version = Interlocked.Increment(ref _version); var vertex = _vertices[index]; - if (vertex == default) { + if (vertex == null) { return version; } _vertices[index] = default; foreach (var incomingIndex in vertex.Incoming) { var incoming = _vertices[incomingIndex]; - if (incoming != default && incoming.IsSealed) { + if (incoming != null && incoming.IsSealed) { _vertices[incomingIndex] = new DependencyVertex(incoming, version, false); } } @@ -86,10 +89,10 @@ public int Remove(in TKey key) { if (!vertex.IsSealed) { return version; } - + foreach (var outgoingIndex in vertex.Outgoing) { var outgoing = _vertices[outgoingIndex]; - if (outgoing != default && !outgoing.IsNew) { + if (outgoing != null && !outgoing.IsNew) { _vertices[outgoingIndex] = new DependencyVertex(outgoing, version, true); } } @@ -116,7 +119,7 @@ public int RemoveKeys(in ImmutableArray keys) { _vertices.Clear(); foreach (var oldVertex in oldVertices) { - if (oldVertex == default) { + if (oldVertex == null) { continue; } @@ -130,7 +133,7 @@ public int RemoveKeys(in ImmutableArray keys) { _keys[key] = index; _vertices.Add(default); } - + Update(key, value, isRoot, incomingKeys, index); } @@ -157,7 +160,7 @@ private ImmutableArray EnsureKeys(in int index, in ImmutableArray key _vertices.Add(default); } else { var vertex = _vertices[keyIndex]; - if (vertex != default && vertex.IsSealed && !vertex.ContainsOutgoing(index)) { + if (vertex != null && vertex.IsSealed && !vertex.ContainsOutgoing(index)) { _vertices[keyIndex] = new DependencyVertex(vertex, version, false); } } @@ -241,7 +244,7 @@ private bool TryBuildReverseGraph(in ImmutableArray[vertices.Count]; foreach (var vertex in vertices) { - if (vertex == default) { + if (vertex == null) { continue; } @@ -250,12 +253,12 @@ private bool TryBuildReverseGraph(in ImmutableArray(); outgoingVertices[incomingIndex] = outgoing; - } + } outgoing.Add(vertex.Index); } @@ -268,7 +271,7 @@ private bool TryBuildReverseGraph(in ImmutableArray>.Create(nodesByVertexIndex.Values); return true; + + bool ReachableFromRoot(ImmutableArray reachable, int index) { + const int inaccessibleFromRoot = -1; + + // one of usage case for this optimization is not analyzing module that is not reachable + // from user code + return _checkVertexReachability && reachable[index] != inaccessibleFromRoot; + } } private static ImmutableArray CalculateDepths(in ImmutableArray> vertices) { @@ -341,7 +352,7 @@ private static ImmutableArray CalculateDepths(in ImmutableArray.Create(depths); } @@ -437,7 +448,7 @@ private bool TryResolveLoops(in ImmutableArray> grap var secondPassVertex = vertex.CreateSecondPassVertex(); var loopNumber = vertex.LoopNumber; if (secondPassLoops[loopNumber] == null) { - secondPassLoops[loopNumber] = new List> {secondPassVertex}; + secondPassLoops[loopNumber] = new List> { secondPassVertex }; } else { secondPassLoops[loopNumber].Add(secondPassVertex); } @@ -533,12 +544,12 @@ private bool TryFindMissingDependencies(in ImmutableArray> _ppc; - public ImmutableArray MissingKeys { get; } + public ImmutableArray MissingKeys { get; set; } public ImmutableArray AffectedValues { get; } public int Version { get; } @@ -690,7 +701,7 @@ private sealed class DependencyChainNode : IDependencyChainNode { public TValue Value => _vertex.DependencyVertex.Value; public int VertexDepth { get; } public bool HasMissingDependencies => _vertex.HasMissingDependencies; - public bool HasOnlyWalkedDependencies => _vertex.HasOnlyWalkedIncoming && _vertex.SecondPass == default; + public bool HasOnlyWalkedDependencies => _vertex.HasOnlyWalkedIncoming && _vertex.SecondPass == null; public bool IsWalkedWithDependencies => _vertex.HasOnlyWalkedIncoming && _vertex.DependencyVertex.IsWalked; public bool IsValidVersion => _walker.IsValidVersion; diff --git a/src/Analysis/Ast/Impl/Dependencies/IDependencyChainWalker.cs b/src/Analysis/Ast/Impl/Dependencies/IDependencyChainWalker.cs index 87697bd27..324aadc5f 100644 --- a/src/Analysis/Ast/Impl/Dependencies/IDependencyChainWalker.cs +++ b/src/Analysis/Ast/Impl/Dependencies/IDependencyChainWalker.cs @@ -19,7 +19,7 @@ namespace Microsoft.Python.Analysis.Dependencies { internal interface IDependencyChainWalker { - ImmutableArray MissingKeys { get; } + ImmutableArray MissingKeys { get; set; } ImmutableArray AffectedValues { get; } int Version { get; } int Remaining { get; } diff --git a/src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs b/src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs index f0e32fd68..499eb484d 100644 --- a/src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs +++ b/src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs @@ -25,18 +25,17 @@ public DiagnosticsSeverityMap() { } public DiagnosticsSeverityMap(string[] errors, string[] warnings, string[] information, string[] disabled) { _map.Clear(); + // disabled > error > warning > information - foreach (var x in information.MaybeEnumerate()) { - _map[x] = Severity.Information; - } - foreach (var x in warnings.MaybeEnumerate()) { - _map[x] = Severity.Warning; - } - foreach (var x in errors.MaybeEnumerate()) { - _map[x] = Severity.Error; - } - foreach (var x in disabled.MaybeEnumerate()) { - _map[x] = Severity.Suppressed; + PopulateMap(information, Severity.Information); + PopulateMap(warnings, Severity.Warning); + PopulateMap(errors, Severity.Error); + PopulateMap(disabled, Severity.Suppressed); + + void PopulateMap(string[] codes, Severity severity) { + foreach (var code in codes.MaybeEnumerate()) { + _map[code] = severity; + } } } public Severity GetEffectiveSeverity(string code, Severity defaultSeverity) diff --git a/src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs b/src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs index cbd7bdb25..a21021c49 100644 --- a/src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs +++ b/src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs @@ -21,6 +21,7 @@ public static class ErrorCodes { public const string ParameterAlreadySpecified = "parameter-already-specified"; public const string ParameterMissing = "parameter-missing"; public const string PositionalArgumentAfterKeyword = "positional-argument-after-keyword"; + public const string PositionalOnlyNamed = "positional-only-named"; public const string ReturnInInit = "return-in-init"; public const string TypingGenericArguments = "typing-generic-arguments"; public const string TypingNewTypeArguments = "typing-newtype-arguments"; diff --git a/src/Analysis/Ast/Impl/Documents/Definitions/IDocument.cs b/src/Analysis/Ast/Impl/Documents/Definitions/IDocument.cs index dc1228ddb..d5b9489ab 100644 --- a/src/Analysis/Ast/Impl/Documents/Definitions/IDocument.cs +++ b/src/Analysis/Ast/Impl/Documents/Definitions/IDocument.cs @@ -63,9 +63,10 @@ public interface IDocument: IPythonModule, IDisposable { void Update(IEnumerable changes); /// - /// Resets document buffer to the provided content or tries to load it if content is null, then parses and analyzes document. + /// Marks document as changed and advances its version. + /// Leads to new parse and analysis of the document. /// - void Reset(string content); + void Invalidate(); /// /// Provides collection of parsing errors, if any. diff --git a/src/Analysis/Ast/Impl/Documents/DocumentBuffer.cs b/src/Analysis/Ast/Impl/Documents/DocumentBuffer.cs index 21a2bdfb8..b7a49ff00 100644 --- a/src/Analysis/Ast/Impl/Documents/DocumentBuffer.cs +++ b/src/Analysis/Ast/Impl/Documents/DocumentBuffer.cs @@ -16,6 +16,7 @@ using System.Collections.Generic; using System.Linq; using System.Text; +using Microsoft.Python.Core.Diagnostics; using Microsoft.Python.Parsing; namespace Microsoft.Python.Analysis.Documents { @@ -23,6 +24,8 @@ internal sealed class DocumentBuffer { private readonly object _lock = new object(); private StringBuilder _sb = new StringBuilder(); private string _content; + private bool _cleared; + private bool _initialized; public int Version { get; private set; } @@ -34,16 +37,44 @@ public string Text { } } - public void Reset(int version, string content) { + public void SetContent(string content) { lock (_lock) { - Version = version; + Check.InvalidOperation(!_initialized, "Buffer is already initialized."); + if (_cleared) { + return; // User may try and edit library file where we have already dropped the content. + } + Version = 0; _content = content ?? string.Empty; _sb = null; + _initialized = true; + } + } + + public void Clear() { + lock (_lock) { + _content = string.Empty; + _sb = null; + _cleared = true; + } + } + + public void MarkChanged() { + lock (_lock) { + Check.InvalidOperation(_initialized, "Buffer is not initialized."); + if (_cleared) { + return; // User may try and edit library file where we have already dropped the content. + } + Version++; } } public void Update(IEnumerable changes) { lock (_lock) { + Check.InvalidOperation(_initialized, "Buffer is not initialized."); + if (_cleared) { + return; // User may try and edit library file where we have already dropped the content. + } + _sb = _sb ?? new StringBuilder(_content); foreach (var change in changes) { @@ -98,7 +129,6 @@ public IEnumerable GetNewLineLocations() { if (i == _sb.Length - 1) { yield return new NewLineLocation(i + 1, NewLineKind.None); } - break; } } diff --git a/src/Analysis/Ast/Impl/Documents/RunningDocumentTable.cs b/src/Analysis/Ast/Impl/Documents/RunningDocumentTable.cs index 810632abc..9b2e259a0 100644 --- a/src/Analysis/Ast/Impl/Documents/RunningDocumentTable.cs +++ b/src/Analysis/Ast/Impl/Documents/RunningDocumentTable.cs @@ -217,7 +217,7 @@ public void ReloadAll() { } foreach (var (_, entry) in opened) { - entry.Document.Reset(null); + entry.Document.Invalidate(); } } @@ -241,7 +241,7 @@ private DocumentEntry CreateDocument(ModuleCreationOptions mco) { IDocument document; switch (mco.ModuleType) { case ModuleType.Compiled when TryAddModulePath(mco): - document = new CompiledPythonModule(mco.ModuleName, ModuleType.Compiled, mco.FilePath, mco.Stub, mco.IsPersistent, _services); + document = new CompiledPythonModule(mco.ModuleName, ModuleType.Compiled, mco.FilePath, mco.Stub, mco.IsPersistent, mco.IsTypeshed, _services); break; case ModuleType.CompiledBuiltin: document = new CompiledBuiltinPythonModule(mco.ModuleName, mco.Stub, mco.IsPersistent, _services); @@ -280,7 +280,6 @@ private bool TryAddModulePath(ModuleCreationOptions mco) { private bool TryOpenDocument(DocumentEntry entry, string content) { if (!entry.Document.IsOpen) { entry.Document.IsOpen = true; - entry.Document.Reset(content); entry.LockCount++; return true; } diff --git a/src/Analysis/Ast/Impl/Extensions/BuiltinTypeIdExtensions.cs b/src/Analysis/Ast/Impl/Extensions/BuiltinTypeIdExtensions.cs index 583dcc124..1d7e65f45 100644 --- a/src/Analysis/Ast/Impl/Extensions/BuiltinTypeIdExtensions.cs +++ b/src/Analysis/Ast/Impl/Extensions/BuiltinTypeIdExtensions.cs @@ -65,7 +65,7 @@ public static string GetTypeName(this BuiltinTypeId id, bool is3x) { case BuiltinTypeId.DictItems: name = "dict_items"; break; case BuiltinTypeId.Function: name = "function"; break; case BuiltinTypeId.Generator: name = "generator"; break; - case BuiltinTypeId.NoneType: name = "NoneType"; break; + case BuiltinTypeId.None: name = "None"; break; case BuiltinTypeId.Ellipsis: name = "ellipsis"; break; case BuiltinTypeId.Module: name = "module_type"; break; case BuiltinTypeId.ListIterator: name = "list_iterator"; break; @@ -114,7 +114,7 @@ public static BuiltinTypeId GetTypeId(this string name) { case "function": return BuiltinTypeId.Function; case "generator": return BuiltinTypeId.Generator; - case "NoneType": return BuiltinTypeId.NoneType; + case "None": return BuiltinTypeId.None; case "ellipsis": return BuiltinTypeId.Ellipsis; case "module_type": return BuiltinTypeId.Module; @@ -143,7 +143,7 @@ internal static PythonMemberType GetMemberId(this BuiltinTypeId id) { case BuiltinTypeId.Long: case BuiltinTypeId.Str: case BuiltinTypeId.Unicode: - case BuiltinTypeId.NoneType: + case BuiltinTypeId.None: case BuiltinTypeId.Ellipsis: case BuiltinTypeId.Dict: case BuiltinTypeId.List: diff --git a/src/Analysis/Ast/Impl/Extensions/MemberExtensions.cs b/src/Analysis/Ast/Impl/Extensions/MemberExtensions.cs index a68f91b51..213dde5d2 100644 --- a/src/Analysis/Ast/Impl/Extensions/MemberExtensions.cs +++ b/src/Analysis/Ast/Impl/Extensions/MemberExtensions.cs @@ -116,5 +116,13 @@ public static bool IsDeclaredAfterOrAt(this ILocatedMember lm, ILocatedMember ot => lm.IsDeclaredAfterOrAt(other.Location); public static bool IsDeclaredAfterOrAt(this ILocatedMember lm, Location loc) => lm.Location.IndexSpan.Start >= loc.IndexSpan.Start; + + public static IPythonFunctionType TryGetFunctionType(this IMember m) { + var t = m.GetPythonType(); + return t is IPythonClassType cls + ? cls.GetMember("__init__") + : t as IPythonFunctionType; + } + } } diff --git a/src/Analysis/Ast/Impl/Extensions/PythonClassExtensions.cs b/src/Analysis/Ast/Impl/Extensions/PythonClassExtensions.cs index da18512ce..7c6d0d1d7 100644 --- a/src/Analysis/Ast/Impl/Extensions/PythonClassExtensions.cs +++ b/src/Analysis/Ast/Impl/Extensions/PythonClassExtensions.cs @@ -17,6 +17,7 @@ using Microsoft.Python.Analysis.Analyzer; using Microsoft.Python.Analysis.Specializations.Typing; using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core; namespace Microsoft.Python.Analysis { @@ -77,5 +78,12 @@ public static bool GetSpecificType(this IPythonClassType cls, string paramName, } return specificType != null; } + + public static bool IsSubClassOf(this IPythonClassType cls, IPythonClassType classToCheck) { + if (classToCheck == null) { + return false; + } + return cls?.Bases.MaybeEnumerate().Any(b => b.Equals(classToCheck)) ?? false; + } } } diff --git a/src/Analysis/Ast/Impl/Extensions/PythonModuleExtensions.cs b/src/Analysis/Ast/Impl/Extensions/PythonModuleExtensions.cs index caa2b5b34..c05e062c3 100644 --- a/src/Analysis/Ast/Impl/Extensions/PythonModuleExtensions.cs +++ b/src/Analysis/Ast/Impl/Extensions/PythonModuleExtensions.cs @@ -13,8 +13,12 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis.Core.DependencyResolution; using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Core; using Microsoft.Python.Core.Text; using Microsoft.Python.Parsing.Ast; @@ -39,12 +43,12 @@ internal static void AddAstNode(this IPythonModule module, object o, Node n) /// /// The line number internal static string GetLine(this IPythonModule module, int lineNum) { - string content = module.Analysis?.Document?.Content; + var content = module.Analysis?.Document?.Content; if (string.IsNullOrEmpty(content)) { return string.Empty; } - SourceLocation source = new SourceLocation(lineNum, 1); + var source = new SourceLocation(lineNum, 1); var start = module.GetAst().LocationToIndex(source); var end = start; @@ -58,9 +62,9 @@ internal static string GetLine(this IPythonModule module, int lineNum) { /// /// The line number internal static string GetComment(this IPythonModule module, int lineNum) { - string line = module.GetLine(lineNum); + var line = module.GetLine(lineNum); - int commentPos = line.IndexOf('#'); + var commentPos = line.IndexOf('#'); if (commentPos < 0) { return string.Empty; } @@ -68,7 +72,8 @@ internal static string GetComment(this IPythonModule module, int lineNum) { return line.Substring(commentPos + 1).Trim('\t', ' '); } - internal static bool IsNonUserFile(this IPythonModule module) => module.ModuleType.IsNonUserFile(); - internal static bool IsCompiled(this IPythonModule module) => module.ModuleType.IsCompiled(); + public static bool IsNonUserFile(this IPythonModule module) => module.ModuleType.IsNonUserFile(); + public static bool IsCompiled(this IPythonModule module) => module.ModuleType.IsCompiled(); + public static bool IsTypingModule(this IPythonModule module) => module?.ModuleType == ModuleType.Specialized && module.Name == "typing"; } } diff --git a/src/Analysis/Ast/Impl/Linting/UndefinedVariables/ExpressionWalker.cs b/src/Analysis/Ast/Impl/Linting/UndefinedVariables/ExpressionWalker.cs index 128511165..011fe16f2 100644 --- a/src/Analysis/Ast/Impl/Linting/UndefinedVariables/ExpressionWalker.cs +++ b/src/Analysis/Ast/Impl/Linting/UndefinedVariables/ExpressionWalker.cs @@ -20,6 +20,7 @@ using Microsoft.Python.Core; using Microsoft.Python.Core.Text; using Microsoft.Python.Parsing.Ast; +using Microsoft.Python.Parsing.Extensions; namespace Microsoft.Python.Analysis.Linting.UndefinedVariables { internal sealed class ExpressionWalker : PythonWalker { @@ -126,8 +127,7 @@ public override bool Walk(NameExpression node) { private bool IsSpanInComprehension(SourceSpan span) { var start = span.Start.ToIndex(_walker.Analysis.Ast); var end = span.End.ToIndex(_walker.Analysis.Ast); - return ((Node)_walker.Analysis.ExpressionEvaluator.CurrentScope.Node) - .TraverseDepthFirst(n => n.GetChildNodes()) + return _walker.Analysis.ExpressionEvaluator.CurrentScope.Node.ChildNodesDepthFirst() .OfType() .Any(n => n.StartIndex <= start && end < n.EndIndex); } diff --git a/src/Analysis/Ast/Impl/Linting/UndefinedVariables/UndefinedVariablesWalker.cs b/src/Analysis/Ast/Impl/Linting/UndefinedVariables/UndefinedVariablesWalker.cs index 7365bf08d..42ac73bf0 100644 --- a/src/Analysis/Ast/Impl/Linting/UndefinedVariables/UndefinedVariablesWalker.cs +++ b/src/Analysis/Ast/Impl/Linting/UndefinedVariables/UndefinedVariablesWalker.cs @@ -49,9 +49,7 @@ public override bool Walk(SuiteStatement node) { HandleNonLocal(nls); break; case AugmentedAssignStatement augs: - _suppressDiagnostics = true; augs.Left?.Walk(new ExpressionWalker(this)); - _suppressDiagnostics = false; augs.Right?.Walk(new ExpressionWalker(this)); break; case AssignmentStatement asst: diff --git a/src/Analysis/Ast/Impl/Microsoft.Python.Analysis.csproj b/src/Analysis/Ast/Impl/Microsoft.Python.Analysis.csproj index 8f4bbee5a..9403083e9 100644 --- a/src/Analysis/Ast/Impl/Microsoft.Python.Analysis.csproj +++ b/src/Analysis/Ast/Impl/Microsoft.Python.Analysis.csproj @@ -9,7 +9,7 @@ 1701, 1702 - "You may need to supply assembly policy" --> 1701;1702;$(NoWarn) - 7.2 + true @@ -18,7 +18,7 @@ all runtime; build; native; contentfiles; analyzers - + diff --git a/src/Analysis/Ast/Impl/Modules/BuiltinsPythonModule.cs b/src/Analysis/Ast/Impl/Modules/BuiltinsPythonModule.cs index 72988f5a2..c8caf615c 100644 --- a/src/Analysis/Ast/Impl/Modules/BuiltinsPythonModule.cs +++ b/src/Analysis/Ast/Impl/Modules/BuiltinsPythonModule.cs @@ -15,14 +15,19 @@ using System; using System.Collections.Generic; +using System.Diagnostics; using System.Linq; +using System.Threading; +using Microsoft.Python.Analysis.Analyzer; using Microsoft.Python.Analysis.Specializations; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core; using Microsoft.Python.Core.IO; +using Microsoft.Python.Core.Logging; using Microsoft.Python.Parsing; using Microsoft.Python.Parsing.Ast; +using Type = Microsoft.Python.Analysis.Specializations.Type; namespace Microsoft.Python.Analysis.Modules { /// @@ -35,29 +40,50 @@ internal sealed class BuiltinsPythonModule : CompiledPythonModule, IBuiltinsPyth private IPythonType _boolType; public BuiltinsPythonModule(string moduleName, string filePath, IServiceContainer services) - : base(moduleName, ModuleType.Builtins, filePath, null, false, services) { } // TODO: builtins stub & persistence + : base(moduleName, ModuleType.Builtins, filePath, null, false, false, services) { } // TODO: builtins stub & persistence + #region IMemberContainer public override IMember GetMember(string name) => _hiddenNames.Contains(name) ? null : base.GetMember(name); public IMember GetAnyMember(string name) => base.GetMember(name); public override IEnumerable GetMemberNames() => base.GetMemberNames().Except(_hiddenNames).ToArray(); + #endregion + + public void Initialize() => ParseAndLogExceptions(CancellationToken.None); protected override string[] GetScrapeArguments(IPythonInterpreter interpreter) => !InstallPath.TryGetFile("scrape_module.py", out var sb) ? null : new[] { "-W", "ignore", "-B", "-E", sb }; + protected override void Parse() { } + + protected override void Analyze(PythonAst ast, int version) { + NotifyAnalysisBegins(); + var walker = Analyze(ast); + var analysis = new DocumentAnalysis(this, version, walker.GlobalScope, walker.Eval, walker.StarImportMemberNames); + NotifyAnalysisComplete(analysis); + } + protected override void OnAnalysisComplete() { SpecializeTypes(); SpecializeFunctions(); foreach (var n in GetMemberNames()) { GetMember(n).GetPythonType()?.MakeReadOnly(); } - base.OnAnalysisComplete(); } + protected override string LoadContent() { + var content = base.LoadContent(); + if (string.IsNullOrEmpty(content)) { + const string message = "Unable continue, no builtins module content."; + Services.GetService()?.Log(TraceEventType.Error, message); + throw new InvalidOperationException(message); + } + return content; + } + private void SpecializeTypes() { - IPythonType noneType = null; var isV3 = Interpreter.LanguageVersion.Is3x(); foreach (BuiltinTypeId typeId in Enum.GetValues(typeof(BuiltinTypeId))) { @@ -112,9 +138,6 @@ private void SpecializeTypes() { case BuiltinTypeId.Bool: _boolType = _boolType ?? biType; break; - case BuiltinTypeId.NoneType: - noneType = noneType ?? biType; - break; } break; } @@ -128,9 +151,7 @@ private void SpecializeTypes() { Analysis.GlobalScope.DeclareVariable("False", _boolType, VariableSource.Builtin, location); } - if (noneType != null) { - Analysis.GlobalScope.DeclareVariable("None", noneType, VariableSource.Builtin, location); - } + Analysis.GlobalScope.DeclareVariable("None", new PythonNone(this), VariableSource.Builtin, location); foreach (var n in GetMemberNames()) { var t = GetMember(n).GetPythonType(); @@ -166,12 +187,13 @@ private void SpecializeFunctions() { Analysis.SpecializeFunction("pow", BuiltinsSpecializations.Identity); Analysis.SpecializeFunction("range", BuiltinsSpecializations.Range); Analysis.SpecializeFunction("sum", BuiltinsSpecializations.CollectionItem); - Analysis.SpecializeFunction("type", BuiltinsSpecializations.TypeInfo); Analysis.SpecializeFunction("vars", BuiltinsSpecializations.DictStringToObject); + Analysis.SpecializeFunction("super", BuiltinsSpecializations.Super); //SpecializeFunction(_builtinName, "range", RangeConstructor); //SpecializeFunction(_builtinName, "sorted", ReturnsListOfInputIterable); - //SpecializeFunction(_builtinName, "super", SpecialSuper); + + Analysis.GlobalScope.DeclareVariable("type", new Type(Analysis.Document), VariableSource.Builtin); } private IReadOnlyList OpenConstructor() { @@ -181,18 +203,18 @@ private IReadOnlyList OpenConstructor() { new ParameterInfo("mode", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant("r", Interpreter.GetBuiltinType(BuiltinTypeId.Str))), new ParameterInfo("buffering", Interpreter.GetBuiltinType(BuiltinTypeId.Int), ParameterKind.Normal, new PythonConstant(-1, Interpreter.GetBuiltinType(BuiltinTypeId.Int))), }; - } else { - return new[] { - new ParameterInfo("file", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, null), - new ParameterInfo("mode", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant("r", Interpreter.GetBuiltinType(BuiltinTypeId.Str))), - new ParameterInfo("buffering", Interpreter.GetBuiltinType(BuiltinTypeId.Int), ParameterKind.Normal, new PythonConstant(-1, Interpreter.GetBuiltinType(BuiltinTypeId.Int))), - new ParameterInfo("encoding", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.NoneType))), - new ParameterInfo("errors", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.NoneType))), - new ParameterInfo("newline", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.NoneType))), - new ParameterInfo("closefd", Interpreter.GetBuiltinType(BuiltinTypeId.Bool), ParameterKind.Normal, new PythonConstant(true, Interpreter.GetBuiltinType(BuiltinTypeId.Bool))), - new ParameterInfo("opener", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.Str))) - }; } + + return new[] { + new ParameterInfo("file", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, null), + new ParameterInfo("mode", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant("r", Interpreter.GetBuiltinType(BuiltinTypeId.Str))), + new ParameterInfo("buffering", Interpreter.GetBuiltinType(BuiltinTypeId.Int), ParameterKind.Normal, new PythonConstant(-1, Interpreter.GetBuiltinType(BuiltinTypeId.Int))), + new ParameterInfo("encoding", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.None))), + new ParameterInfo("errors", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.None))), + new ParameterInfo("newline", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.None))), + new ParameterInfo("closefd", Interpreter.GetBuiltinType(BuiltinTypeId.Bool), ParameterKind.Normal, new PythonConstant(true, Interpreter.GetBuiltinType(BuiltinTypeId.Bool))), + new ParameterInfo("opener", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.Str))) + }; } } } diff --git a/src/Analysis/Ast/Impl/Modules/CompiledBuiltinPythonModule.cs b/src/Analysis/Ast/Impl/Modules/CompiledBuiltinPythonModule.cs index 500e4dbca..bf1e3d0c5 100644 --- a/src/Analysis/Ast/Impl/Modules/CompiledBuiltinPythonModule.cs +++ b/src/Analysis/Ast/Impl/Modules/CompiledBuiltinPythonModule.cs @@ -24,7 +24,7 @@ namespace Microsoft.Python.Analysis.Modules { /// internal sealed class CompiledBuiltinPythonModule : CompiledPythonModule { public CompiledBuiltinPythonModule(string moduleName, IPythonModule stub, bool isPersistent, IServiceContainer services) - : base(moduleName, ModuleType.CompiledBuiltin, MakeFakeFilePath(moduleName, services), stub, isPersistent, services) { } + : base(moduleName, ModuleType.CompiledBuiltin, MakeFakeFilePath(moduleName, services), stub, isPersistent, false, services) { } protected override string[] GetScrapeArguments(IPythonInterpreter interpreter) => !InstallPath.TryGetFile("scrape_module.py", out var sm) ? null : new [] { "-W", "ignore", "-B", "-E", sm, "-u8", Name }; diff --git a/src/Analysis/Ast/Impl/Modules/CompiledPythonModule.cs b/src/Analysis/Ast/Impl/Modules/CompiledPythonModule.cs index 662b25d30..a5ddf50db 100644 --- a/src/Analysis/Ast/Impl/Modules/CompiledPythonModule.cs +++ b/src/Analysis/Ast/Impl/Modules/CompiledPythonModule.cs @@ -26,8 +26,8 @@ namespace Microsoft.Python.Analysis.Modules { internal class CompiledPythonModule : PythonModule { protected IStubCache StubCache => Interpreter.ModuleResolution.StubCache; - public CompiledPythonModule(string moduleName, ModuleType moduleType, string filePath, IPythonModule stub, bool isPersistent, IServiceContainer services) - : base(moduleName, filePath, moduleType, stub, isPersistent, services) { } + public CompiledPythonModule(string moduleName, ModuleType moduleType, string filePath, IPythonModule stub, bool isPersistent, bool isTypeshed, IServiceContainer services) + : base(moduleName, filePath, moduleType, stub, isPersistent, isTypeshed, services) { } public override string Documentation => GetMember("__doc__").TryGetConstant(out var s) ? s : string.Empty; @@ -100,7 +100,9 @@ private string ScrapeModule() { output = process.StandardOutput.ReadToEnd(); } - } catch (Exception ex) when (!ex.IsCriticalException()) { } + } catch (Exception ex) when (!ex.IsCriticalException()) { + Log?.Log(TraceEventType.Verbose, "Exception scraping module", Name, ex.Message); + } return output; } diff --git a/src/Analysis/Ast/Impl/Modules/Definitions/IModuleManagement.cs b/src/Analysis/Ast/Impl/Modules/Definitions/IModuleManagement.cs index 1f2b0b936..91a034f98 100644 --- a/src/Analysis/Ast/Impl/Modules/Definitions/IModuleManagement.cs +++ b/src/Analysis/Ast/Impl/Modules/Definitions/IModuleManagement.cs @@ -15,6 +15,7 @@ using System; using System.Collections.Generic; +using System.Threading; using Microsoft.Python.Analysis.Caching; using Microsoft.Python.Analysis.Core.Interpreter; using Microsoft.Python.Analysis.Types; @@ -87,5 +88,7 @@ public interface IModuleManagement : IModuleResolution { ImmutableArray LibraryPaths { get; } bool SetUserConfiguredPaths(ImmutableArray paths); + + IEnumerable GetImportedModules(CancellationToken cancellationToken); } } diff --git a/src/Analysis/Ast/Impl/Modules/Definitions/ModuleCreationOptions.cs b/src/Analysis/Ast/Impl/Modules/Definitions/ModuleCreationOptions.cs index f3b1058c8..ff57426cc 100644 --- a/src/Analysis/Ast/Impl/Modules/Definitions/ModuleCreationOptions.cs +++ b/src/Analysis/Ast/Impl/Modules/Definitions/ModuleCreationOptions.cs @@ -52,5 +52,11 @@ public sealed class ModuleCreationOptions { /// Indicates if module is restored from database. /// public bool IsPersistent { get; set; } + + /// + /// Defines if module belongs to Typeshed and hence resolved + /// via typeshed module resolution service. + /// + public bool IsTypeshed { get; set; } } } diff --git a/src/Analysis/Ast/Impl/Modules/Definitions/ModuleState.cs b/src/Analysis/Ast/Impl/Modules/Definitions/ModuleState.cs new file mode 100644 index 000000000..b1bea0428 --- /dev/null +++ b/src/Analysis/Ast/Impl/Modules/Definitions/ModuleState.cs @@ -0,0 +1,26 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +namespace Microsoft.Python.Analysis.Modules { + public enum ModuleState { + None, + Loading, + Loaded, + Parsing, + Parsed, + Analyzing, + Analyzed + } +} diff --git a/src/Analysis/Ast/Impl/Modules/PythonModule.cs b/src/Analysis/Ast/Impl/Modules/PythonModule.cs index af216a813..031cef760 100644 --- a/src/Analysis/Ast/Impl/Modules/PythonModule.cs +++ b/src/Analysis/Ast/Impl/Modules/PythonModule.cs @@ -22,6 +22,8 @@ using System.Threading; using System.Threading.Tasks; using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Analysis.Analyzer.Evaluation; +using Microsoft.Python.Analysis.Analyzer.Handlers; using Microsoft.Python.Analysis.Dependencies; using Microsoft.Python.Analysis.Diagnostics; using Microsoft.Python.Analysis.Documents; @@ -44,16 +46,6 @@ namespace Microsoft.Python.Analysis.Modules { /// [DebuggerDisplay("{Name} : {ModuleType}")] internal class PythonModule : LocatedMember, IDocument, IAnalyzable, IEquatable, IAstNodeContainer, ILocationConverter { - private enum State { - None, - Loading, - Loaded, - Parsing, - Parsed, - Analyzing, - Analyzed - } - private readonly DocumentBuffer _buffer = new DocumentBuffer(); private readonly DisposeToken _disposeToken = DisposeToken.Create(); private readonly object _syncObj = new object(); @@ -70,7 +62,6 @@ private enum State { protected ILogger Log { get; } protected IFileSystem FileSystem { get; } protected IServiceContainer Services { get; } - private State ContentState { get; set; } = State.None; protected PythonModule(string name, ModuleType moduleType, IServiceContainer services) : base(null) { Name = name ?? throw new ArgumentNullException(nameof(name)); @@ -86,12 +77,13 @@ protected PythonModule(string name, ModuleType moduleType, IServiceContainer ser SetDeclaringModule(this); } - protected PythonModule(string moduleName, string filePath, ModuleType moduleType, IPythonModule stub, bool isPersistent, IServiceContainer services) : + protected PythonModule(string moduleName, string filePath, ModuleType moduleType, IPythonModule stub, bool isPersistent, bool isTypeshed, IServiceContainer services) : this(new ModuleCreationOptions { ModuleName = moduleName, FilePath = filePath, ModuleType = moduleType, Stub = stub, + IsTypeshed = isTypeshed, IsPersistent = isPersistent }, services) { } @@ -115,10 +107,12 @@ internal PythonModule(ModuleCreationOptions creationOptions, IServiceContainer s } if (ModuleType == ModuleType.Specialized || ModuleType == ModuleType.Unresolved) { - ContentState = State.Analyzed; + ModuleState = ModuleState.Analyzed; } IsPersistent = creationOptions.IsPersistent; + IsTypeshed = creationOptions.IsTypeshed; + InitializeContent(creationOptions.Content, 0); } @@ -130,7 +124,7 @@ internal PythonModule(ModuleCreationOptions creationOptions, IServiceContainer s public bool IsAbstract => false; public virtual bool IsSpecialized => false; - public IPythonInstance CreateInstance(IArgumentSet args) => new PythonInstance(this); + public IMember CreateInstance(IArgumentSet args) => new PythonInstance(this); public override PythonMemberType MemberType => PythonMemberType.Module; public IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) => GetMember(memberName); public IMember Index(IPythonInstance instance, IArgumentSet args) => Interpreter.UnknownType; @@ -158,6 +152,7 @@ public virtual string Documentation { #region IMemberContainer public virtual IMember GetMember(string name) => GlobalScope.Variables[name]?.Value; + public virtual IEnumerable GetMemberNames() { // drop imported modules and typing. return GlobalScope.Variables @@ -166,26 +161,26 @@ public virtual IEnumerable GetMemberNames() { if (v.Value is IPythonInstance) { return true; } + var valueType = v.Value?.GetPythonType(); - if (valueType is PythonModule) { - return false; // Do not re-export modules. - } - if (valueType is IPythonFunctionType f && f.IsLambda()) { - return false; + switch (valueType) { + case PythonModule _: + case IPythonFunctionType f when f.IsLambda(): + return false; // Do not re-export modules. } + if (this is TypingModule) { return true; // Let typing module behave normally. } + // Do not re-export types from typing. However, do export variables // assigned with types from typing. Example: // from typing import Any # do NOT export Any // x = Union[int, str] # DO export x - if (valueType?.DeclaringModule is TypingModule && v.Name == valueType.Name) { - return false; - } - return true; + return !(valueType?.DeclaringModule is TypingModule) || v.Name != valueType.Name; }) - .Select(v => v.Name); + .Select(v => v.Name) + .ToArray(); } #endregion @@ -198,6 +193,7 @@ public virtual IEnumerable GetMemberNames() { public virtual Uri Uri { get; } public IDocumentAnalysis Analysis { get; private set; } public IPythonInterpreter Interpreter { get; } + public ModuleState ModuleState { get; private set; } = ModuleState.None; /// /// Associated stub module. Note that in case of specialized modules @@ -221,6 +217,12 @@ public virtual IEnumerable GetMemberNames() { /// Indicates if module is restored from database. /// public bool IsPersistent { get; } + + /// + /// Defines if module belongs to Typeshed and hence resolved + /// via typeshed module resolution service. + /// + public bool IsTypeshed { get; } #endregion #region IDisposable @@ -310,30 +312,45 @@ public void Update(IEnumerable changes) { Parse(); } - Services.GetService().InvalidateAnalysis(this); } - public void Reset(string content) { + public void Invalidate() { lock (_syncObj) { - if (content != Content) { - ContentState = State.None; - InitializeContent(content, _buffer.Version + 1); - } + ModuleState = ModuleState.None; + _buffer.MarkChanged(); + Parse(); } - Services.GetService().InvalidateAnalysis(this); } - private void Parse() { + protected virtual void Parse() { _parseCts?.Cancel(); _parseCts = new CancellationTokenSource(); _linkedParseCts?.Dispose(); _linkedParseCts = CancellationTokenSource.CreateLinkedTokenSource(_disposeToken.CancellationToken, _parseCts.Token); - ContentState = State.Parsing; - _parsingTask = Task.Run(() => Parse(_linkedParseCts.Token), _linkedParseCts.Token); + ModuleState = ModuleState.Parsing; + _parsingTask = Task.Run(() => ParseAndLogExceptions(_linkedParseCts.Token), _linkedParseCts.Token); + } + + protected void ParseAndLogExceptions(CancellationToken cancellationToken) { + try { + Parse(cancellationToken); + } catch (Exception ex) when (!(ex is OperationCanceledException)) { + Log?.Log(TraceEventType.Warning, $"Exception while parsing {FilePath}: {ex}"); + throw; + } + } + + protected virtual void Analyze(PythonAst ast, int version) { + if (ModuleState < ModuleState.Analyzing) { + ModuleState = ModuleState.Analyzing; + + var analyzer = Services.GetService(); + analyzer.EnqueueDocumentForAnalysis(this, ast, version); + } } private void Parse(CancellationToken cancellationToken) { @@ -356,8 +373,7 @@ private void Parse(CancellationToken cancellationToken) { } var ast = parser.ParseFile(Uri); - - //Log?.Log(TraceEventType.Verbose, $"Parse complete: {Name}"); + // Log?.Log(TraceEventType.Verbose, $"Parse complete: {Name} ({ModuleType})"); lock (_syncObj) { cancellationToken.ThrowIfCancellationRequested(); @@ -376,18 +392,12 @@ private void Parse(CancellationToken cancellationToken) { _diagnosticsService?.Replace(Uri, _parseErrors, DiagnosticSource.Parser); } - ContentState = State.Parsed; + ModuleState = ModuleState.Parsed; Analysis = new EmptyAnalysis(Services, this); } NewAst?.Invoke(this, EventArgs.Empty); - - if (ContentState < State.Analyzing) { - ContentState = State.Analyzing; - - var analyzer = Services.GetService(); - analyzer.EnqueueDocumentForAnalysis(this, ast, version); - } + Analyze(ast, version); lock (_syncObj) { _parsingTask = null; @@ -434,6 +444,13 @@ public void NotifyAnalysisBegins() { } } + public ModuleWalker Analyze(PythonAst ast) { + var walker = new ModuleWalker(Services, this, ast, CancellationToken.None); + ast.Walk(walker); + walker.Complete(); + return walker; + } + public void NotifyAnalysisComplete(IDocumentAnalysis analysis) { lock (_syncObj) { if (analysis.Version < Analysis.Version) { @@ -447,10 +464,10 @@ public void NotifyAnalysisComplete(IDocumentAnalysis analysis) { // to perform additional actions on the completed analysis such // as declare additional variables, etc. OnAnalysisComplete(); - ContentState = State.Analyzed; + ModuleState = ModuleState.Analyzed; if (ModuleType != ModuleType.User) { - _buffer.Reset(_buffer.Version, string.Empty); + _buffer.Clear(); } } @@ -486,7 +503,7 @@ public void AddAstNode(object o, Node n) { public void ClearContent() { lock (_syncObj) { if (ModuleType != ModuleType.User) { - _buffer.Reset(_buffer.Version, string.Empty); + _buffer.Clear(); _astMap.Clear(); } } @@ -503,11 +520,11 @@ public Task GetAnalysisAsync(int waitTime = 200, Cancellation #region Content management protected virtual string LoadContent() { - if (ContentState < State.Loading) { - ContentState = State.Loading; + if (ModuleState < ModuleState.Loading) { + ModuleState = ModuleState.Loading; try { var code = FileSystem.ReadTextWithRetry(FilePath); - ContentState = State.Loaded; + ModuleState = ModuleState.Loaded; return code; } catch (IOException) { } catch (UnauthorizedAccessException) { } } @@ -516,25 +533,24 @@ protected virtual string LoadContent() { private void InitializeContent(string content, int version) { lock (_syncObj) { - LoadContent(content, version); - - var startParse = ContentState < State.Parsing && (_parsingTask == null || version > 0); - if (startParse) { + SetOrLoadContent(content); + if (ModuleState < ModuleState.Parsing && _parsingTask == null) { Parse(); } } + Services.GetService().InvalidateAnalysis(this); } - private void LoadContent(string content, int version) { - if (ContentState < State.Loading) { + private void SetOrLoadContent(string content) { + if (ModuleState < ModuleState.Loading) { try { if (IsPersistent) { content = string.Empty; } else { content = content ?? LoadContent(); } - _buffer.Reset(version, content); - ContentState = State.Loaded; + _buffer.SetContent(content); + ModuleState = ModuleState.Loaded; } catch (IOException) { } catch (UnauthorizedAccessException) { } } } diff --git a/src/Analysis/Ast/Impl/Modules/PythonVariableModule.cs b/src/Analysis/Ast/Impl/Modules/PythonVariableModule.cs index 70e4c9fab..51550546f 100644 --- a/src/Analysis/Ast/Impl/Modules/PythonVariableModule.cs +++ b/src/Analysis/Ast/Impl/Modules/PythonVariableModule.cs @@ -37,6 +37,7 @@ internal sealed class PythonVariableModule : LocatedMember, IPythonModule, IEqua public IPythonModule Module { get; } public IPythonInterpreter Interpreter { get; } + public ModuleState ModuleState => Module?.ModuleState ?? ModuleState.None; public IDocumentAnalysis Analysis => Module?.Analysis; public string Documentation => Module?.Documentation ?? string.Empty; @@ -52,6 +53,7 @@ internal sealed class PythonVariableModule : LocatedMember, IPythonModule, IEqua public Uri Uri => Module?.Uri; public override PythonMemberType MemberType => PythonMemberType.Module; public bool IsPersistent => Module?.IsPersistent == true; + public bool IsTypeshed => Module?.IsTypeshed == true; public PythonVariableModule(string name, IPythonInterpreter interpreter) : base(null) { Name = name; @@ -67,15 +69,18 @@ public PythonVariableModule(IPythonModule module): base(module) { public void AddChildModule(string memberName, PythonVariableModule module) => _children[memberName] = module; - public IMember GetMember(string name) => Module?.GetMember(name) ?? (_children.TryGetValue(name, out var module) ? module : default); - public IEnumerable GetMemberNames() => Module != null ? Module.GetMemberNames().Concat(_children.Keys) : _children.Keys; + public IMember GetMember(string name) => _children.TryGetValue(name, out var module) ? module : Module?.GetMember(name); + public IEnumerable GetMemberNames() => Module != null ? Module.GetMemberNames().Concat(_children.Keys).Distinct() : _children.Keys; public IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) => GetMember(memberName); public IMember Index(IPythonInstance instance, IArgumentSet args) => Interpreter.UnknownType; - public IPythonInstance CreateInstance(IArgumentSet args = null) => new PythonInstance(this); + public IMember CreateInstance(IArgumentSet args = null) => new PythonInstance(this); public bool Equals(IPythonModule other) => other is PythonVariableModule module && Name.EqualsOrdinal(module.Name); + public override bool Equals(object obj) => Equals(obj as IPythonModule); + public override int GetHashCode() => 0; + #region ILocationConverter public SourceLocation IndexToLocation(int index) => (Module as ILocationConverter)?.IndexToLocation(index) ?? default; public int LocationToIndex(SourceLocation location) => (Module as ILocationConverter)?.LocationToIndex(location) ?? default; diff --git a/src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs b/src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs index 26c0d3f83..22dafdadd 100644 --- a/src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs +++ b/src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs @@ -39,6 +39,7 @@ namespace Microsoft.Python.Analysis.Modules.Resolution { internal sealed class MainModuleResolution : ModuleResolutionBase, IModuleManagement { private readonly ConcurrentDictionary _specialized = new ConcurrentDictionary(); private readonly IUIService _ui; + private BuiltinsPythonModule _builtins; private IModuleDatabaseService _dbService; private IRunningDocumentTable _rdt; @@ -53,7 +54,21 @@ public MainModuleResolution(string root, IServiceContainer services, ImmutableAr public string BuiltinModuleName => BuiltinTypeId.Unknown.GetModuleName(Interpreter.LanguageVersion); public ImmutableArray LibraryPaths { get; private set; } = ImmutableArray.Empty; - public IBuiltinsPythonModule BuiltinsModule { get; private set; } + public IBuiltinsPythonModule BuiltinsModule => _builtins; + + public IEnumerable GetImportedModules(CancellationToken cancellationToken) { + foreach (var module in _specialized.Values) { + cancellationToken.ThrowIfCancellationRequested(); + yield return module; + } + + foreach (var moduleRef in Modules.Values) { + cancellationToken.ThrowIfCancellationRequested(); + if (moduleRef.Value != null) { + yield return moduleRef.Value; + } + } + } protected override IPythonModule CreateModule(string name) { var moduleImport = CurrentPathResolver.GetModuleImportFromModuleName(name); @@ -95,7 +110,7 @@ protected override IPythonModule CreateModule(string name) { if (moduleImport.IsCompiled) { Log?.Log(TraceEventType.Verbose, "Create compiled (scraped): ", moduleImport.FullName, moduleImport.ModulePath, moduleImport.RootPath); - return new CompiledPythonModule(moduleImport.FullName, ModuleType.Compiled, moduleImport.ModulePath, stub, moduleImport.IsPersistent, Services); + return new CompiledPythonModule(moduleImport.FullName, ModuleType.Compiled, moduleImport.ModulePath, stub, moduleImport.IsPersistent, false, Services); } Log?.Log(TraceEventType.Verbose, "Import: ", moduleImport.FullName, moduleImport.ModulePath); @@ -164,12 +179,8 @@ public IPythonModule GetSpecializedModule(string fullName, bool allowCreation = public bool IsSpecializedModule(string fullName, string modulePath = null) => _specialized.ContainsKey(fullName); - internal async Task AddBuiltinTypesToPathResolverAsync(CancellationToken cancellationToken = default) { - var analyzer = Services.GetService(); - await analyzer.GetAnalysisAsync(BuiltinsModule, -1, cancellationToken); - - Check.InvalidOperation(!(BuiltinsModule.Analysis is EmptyAnalysis), "After await"); - + private void AddBuiltinTypesToPathResolver() { + Check.InvalidOperation(!(BuiltinsModule.Analysis is EmptyAnalysis), "Builtins analysis did not complete correctly."); // Add built-in module names var builtinModuleNamesMember = BuiltinsModule.GetAnyMember("__builtin_module_names__"); var value = (builtinModuleNamesMember as IVariable)?.Value ?? builtinModuleNamesMember; @@ -198,19 +209,19 @@ public async Task ReloadAsync(CancellationToken cancellationToken = default) { var addedRoots = new HashSet { Root }; addedRoots.UnionWith(InterpreterPaths); addedRoots.UnionWith(UserPaths); - ReloadModulePaths(addedRoots); + ReloadModulePaths(addedRoots, cancellationToken); if (!builtinsIsCreated) { - var builtinsModule = CreateBuiltinsModule(Services, Interpreter, StubCache); - BuiltinsModule = builtinsModule; - builtinsRef = new ModuleRef(builtinsModule); + _builtins = CreateBuiltinsModule(Services, Interpreter, StubCache); + builtinsRef = new ModuleRef(_builtins); + _builtins.Initialize(); } Modules[BuiltinModuleName] = builtinsRef; - await AddBuiltinTypesToPathResolverAsync(cancellationToken); + AddBuiltinTypesToPathResolver(); } - private static IBuiltinsPythonModule CreateBuiltinsModule(IServiceContainer services, IPythonInterpreter interpreter, IStubCache stubCache) { + private static BuiltinsPythonModule CreateBuiltinsModule(IServiceContainer services, IPythonInterpreter interpreter, IStubCache stubCache) { var moduleName = BuiltinTypeId.Unknown.GetModuleName(interpreter.LanguageVersion); var modulePath = stubCache.GetCacheFilePath(interpreter.Configuration.InterpreterPath); return new BuiltinsPythonModule(moduleName, modulePath, services); diff --git a/src/Analysis/Ast/Impl/Modules/Resolution/ModuleResolutionBase.cs b/src/Analysis/Ast/Impl/Modules/Resolution/ModuleResolutionBase.cs index 64c6d5115..0a3782b24 100644 --- a/src/Analysis/Ast/Impl/Modules/Resolution/ModuleResolutionBase.cs +++ b/src/Analysis/Ast/Impl/Modules/Resolution/ModuleResolutionBase.cs @@ -17,6 +17,7 @@ using System.Collections.Generic; using System.IO; using System.Linq; +using System.Threading; using Microsoft.Python.Analysis.Caching; using Microsoft.Python.Analysis.Core.DependencyResolution; using Microsoft.Python.Analysis.Core.Interpreter; @@ -62,8 +63,14 @@ protected ModuleResolutionBase(string root, IServiceContainer services) { protected abstract IPythonModule CreateModule(string name); - public IPythonModule GetImportedModule(string name) - => Modules.TryGetValue(name, out var moduleRef) ? moduleRef.Value : Interpreter.ModuleResolution.GetSpecializedModule(name); + public IPythonModule GetImportedModule(string name) { + if (name == Interpreter.ModuleResolution.BuiltinsModule.Name) { + return Interpreter.ModuleResolution.BuiltinsModule; + } + return Modules.TryGetValue(name, out var moduleRef) + ? moduleRef.Value + : Interpreter.ModuleResolution.GetSpecializedModule(name); + } public IPythonModule GetOrLoadModule(string name) { // Specialized should always win. However, we don't want @@ -74,11 +81,6 @@ public IPythonModule GetOrLoadModule(string name) { return module; } - module = Interpreter.ModuleResolution.GetSpecializedModule(name); - if (module != null) { - return module; - } - // Now try regular case. if (Modules.TryGetValue(name, out var moduleRef)) { return moduleRef.GetOrCreate(name, this); @@ -101,19 +103,21 @@ public ModulePath FindModule(string filePath) { return ModulePath.FromFullPath(filePath, bestLibraryPath); } - protected void ReloadModulePaths(in IEnumerable rootPaths) { + protected void ReloadModulePaths(in IEnumerable rootPaths, CancellationToken cancellationToken) { foreach (var root in rootPaths) { foreach (var moduleFile in PathUtils.EnumerateFiles(FileSystem, root)) { - PathResolver.TryAddModulePath(moduleFile.FullName, moduleFile.Length, false, out _); + cancellationToken.ThrowIfCancellationRequested(); + PathResolver.TryAddModulePath(moduleFile.FullName, moduleFile.Length, allowNonRooted: false, out _); } if (PathUtils.TryGetZipFilePath(root, out var zipFilePath, out var _) && File.Exists(zipFilePath)) { foreach (var moduleFile in PathUtils.EnumerateZip(zipFilePath)) { + cancellationToken.ThrowIfCancellationRequested(); if (!PathUtils.PathStartsWith(moduleFile.FullName, "EGG-INFO")) { PathResolver.TryAddModulePath( Path.Combine(zipFilePath, PathUtils.NormalizePath(moduleFile.FullName)), - moduleFile.Length, false, out _ + moduleFile.Length, allowNonRooted: false, out _ ); } } diff --git a/src/Analysis/Ast/Impl/Modules/Resolution/TypeshedResolution.cs b/src/Analysis/Ast/Impl/Modules/Resolution/TypeshedResolution.cs index af3ad1884..971fe86c5 100644 --- a/src/Analysis/Ast/Impl/Modules/Resolution/TypeshedResolution.cs +++ b/src/Analysis/Ast/Impl/Modules/Resolution/TypeshedResolution.cs @@ -32,8 +32,11 @@ internal sealed class TypeshedResolution : ModuleResolutionBase, IModuleResoluti public TypeshedResolution(string root, IServiceContainer services) : base(root, services) { // TODO: merge with user-provided stub paths - var stubs = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "Stubs"); - _typeStubPaths = GetTypeShedPaths(Root) + var asmLocation = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location); + var stubs = Path.Combine(asmLocation, "Stubs"); + var typeshedRoot = Root ?? Path.Combine(asmLocation, "Typeshed"); + + _typeStubPaths = GetTypeShedPaths(typeshedRoot) .Concat(GetTypeShedPaths(stubs)) .Where(services.GetService().DirectoryExists) .ToImmutableArray(); @@ -46,7 +49,7 @@ public TypeshedResolution(string root, IServiceContainer services) : base(root, protected override IPythonModule CreateModule(string name) { var moduleImport = CurrentPathResolver.GetModuleImportFromModuleName(name); - if (moduleImport != default) { + if (moduleImport != null) { if (moduleImport.IsCompiled) { Log?.Log(TraceEventType.Warning, "Unsupported native module in stubs", moduleImport.FullName, moduleImport.ModulePath); return null; @@ -67,7 +70,7 @@ protected override IPythonModule CreateModule(string name) { public Task ReloadAsync(CancellationToken cancellationToken = default) { Modules.Clear(); PathResolver = new PathResolver(Interpreter.LanguageVersion, Root, _typeStubPaths, ImmutableArray.Empty); - ReloadModulePaths(_typeStubPaths.Prepend(Root)); + ReloadModulePaths(_typeStubPaths.Prepend(Root), cancellationToken); cancellationToken.ThrowIfCancellationRequested(); return Task.CompletedTask; } diff --git a/src/Analysis/Ast/Impl/Modules/SpecializedModule.cs b/src/Analysis/Ast/Impl/Modules/SpecializedModule.cs index 771acd25f..f9f545348 100644 --- a/src/Analysis/Ast/Impl/Modules/SpecializedModule.cs +++ b/src/Analysis/Ast/Impl/Modules/SpecializedModule.cs @@ -32,7 +32,7 @@ namespace Microsoft.Python.Analysis.Modules { /// internal abstract class SpecializedModule : PythonModule { protected SpecializedModule(string name, string modulePath, IServiceContainer services) - : base(name, modulePath, ModuleType.Specialized, null, false, services) { } + : base(name, modulePath, ModuleType.Specialized, null, false, false, services) { } protected override string LoadContent() { // Exceptions are handled in the base diff --git a/src/Analysis/Ast/Impl/Modules/StubPythonModule.cs b/src/Analysis/Ast/Impl/Modules/StubPythonModule.cs index 862daf817..7709c36c0 100644 --- a/src/Analysis/Ast/Impl/Modules/StubPythonModule.cs +++ b/src/Analysis/Ast/Impl/Modules/StubPythonModule.cs @@ -22,11 +22,8 @@ namespace Microsoft.Python.Analysis.Modules { /// Represents module that contains stub code such as from typeshed. /// internal class StubPythonModule : CompiledPythonModule { - public bool IsTypeshed { get; } - public StubPythonModule(string moduleName, string stubPath, bool isTypeshed, IServiceContainer services) - : base(moduleName, ModuleType.Stub, stubPath, null, false, services) { - IsTypeshed = isTypeshed; + : base(moduleName, ModuleType.Stub, stubPath, null, false, isTypeshed, services) { } protected override string LoadContent() { diff --git a/src/Analysis/Ast/Impl/Resources.Designer.cs b/src/Analysis/Ast/Impl/Resources.Designer.cs index c55b49b5f..99dc7ba07 100644 --- a/src/Analysis/Ast/Impl/Resources.Designer.cs +++ b/src/Analysis/Ast/Impl/Resources.Designer.cs @@ -87,6 +87,15 @@ internal static string Analysis_PositionalArgumentAfterKeyword { } } + /// + /// Looks up a localized string similar to Positional only argument '{0}' may not be named.. + /// + internal static string Analysis_PositionalOnlyArgumentNamed { + get { + return ResourceManager.GetString("Analysis_PositionalOnlyArgumentNamed", resourceCulture); + } + } + /// /// Looks up a localized string similar to Too many function arguments.. /// @@ -258,6 +267,15 @@ internal static string InterpreterNotFound { } } + /// + /// Looks up a localized string similar to Specified cache folder ('{0}') does not exist. Switching to default.. + /// + internal static string Invalid_0_CacheFolder { + get { + return ResourceManager.GetString("Invalid_0_CacheFolder", resourceCulture); + } + } + /// /// Looks up a localized string similar to The first argument to NewType must be a string.. /// diff --git a/src/Analysis/Ast/Impl/Resources.resx b/src/Analysis/Ast/Impl/Resources.resx index d95a64109..b7e1a9424 100644 --- a/src/Analysis/Ast/Impl/Resources.resx +++ b/src/Analysis/Ast/Impl/Resources.resx @@ -219,4 +219,10 @@ Method '{0}' has no argument. + + Positional only argument '{0}' may not be named. + + + Specified cache folder ('{0}') does not exist. Switching to default. + \ No newline at end of file diff --git a/src/Analysis/Ast/Impl/Specializations/BuiltinsSpecializations.cs b/src/Analysis/Ast/Impl/Specializations/BuiltinsSpecializations.cs index 5a1b2283c..4ae90aca6 100644 --- a/src/Analysis/Ast/Impl/Specializations/BuiltinsSpecializations.cs +++ b/src/Analysis/Ast/Impl/Specializations/BuiltinsSpecializations.cs @@ -21,6 +21,7 @@ using Microsoft.Python.Analysis.Values; using Microsoft.Python.Analysis.Values.Collections; using Microsoft.Python.Core.Text; +using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.Analysis.Specializations { public static class BuiltinsSpecializations { @@ -29,12 +30,6 @@ public static IMember Identity(IPythonModule module, IPythonFunctionOverload ove return args.Count > 0 ? args.FirstOrDefault(a => !a.IsUnknown()) ?? args[0] : null; } - public static IMember TypeInfo(IPythonModule module, IPythonFunctionOverload overload, IArgumentSet argSet, IndexSpan indexSpan) { - var args = argSet.Values(); - var t = args.Count > 0 ? args[0].GetPythonType() : module.Interpreter.GetBuiltinType(BuiltinTypeId.Type); - return t.ToBound(); - } - public static IMember Iterator(IPythonModule module, IPythonFunctionOverload overload, IArgumentSet argSet, IndexSpan indexSpan) { var args = argSet.Values(); if (args.Count > 0) { @@ -91,6 +86,40 @@ public static IMember CollectionItem(IPythonModule module, IPythonFunctionOverlo return args.Count > 0 && args[0] is PythonCollection c ? c.Contents.FirstOrDefault() : null; } + public static IMember Super(IPythonModule declaringModule, IPythonFunctionOverload overload, IArgumentSet argSet, IndexSpan indexSpan) { + var args = argSet.Values(); + + if (args.Count == 0) { + //Zero argument form only works inside a class definition + foreach (var s in argSet.Eval.CurrentScope.EnumerateTowardsGlobal.Where(s => s.Node is ClassDefinition)) { + var classType = s.Variables["__class__"].GetPythonType(); + return PythonSuperType.Create(classType)?.CreateInstance(argSet); + } + return null; + } + + // If multiple arguments first argument is required + var firstCls = args.FirstOrDefault().GetPythonType(); + if (firstCls == null) { + return null; + } + + // second argument optional + bool isUnbound = args.Count == 1; + if (isUnbound) { + return PythonSuperType.Create(firstCls)?.CreateInstance(argSet); + } + + var secondCls = args[1].GetPythonType(); + if (secondCls?.Equals(firstCls) == true || + secondCls?.IsSubClassOf(firstCls) == true) { + // We walk the mro of the second parameter looking for the first + return PythonSuperType.Create(secondCls, typeToFind: firstCls)?.CreateInstance(argSet); + } + + return null; + } + public static IMember Open(IPythonModule declaringModule, IPythonFunctionOverload overload, IArgumentSet argSet, IndexSpan indexSpan) { var mode = argSet.GetArgumentValue("mode"); @@ -141,7 +170,7 @@ public static IMember GetAttr(IPythonModule module, IPythonFunctionOverload over // getattr(a, 3.14) if (name == null) { // TODO diagnostic error when second arg of getattr is not a string - return module.Interpreter.UnknownType; + return module.Interpreter.UnknownType; } return o?.GetPythonType().GetMember(name) ?? def; diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Definitions/ITypingNamedTupleType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Definitions/ITypingNamedTupleType.cs index bc22e2a3a..61f35325a 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Definitions/ITypingNamedTupleType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Definitions/ITypingNamedTupleType.cs @@ -21,5 +21,16 @@ namespace Microsoft.Python.Analysis.Specializations.Typing { /// public interface ITypingNamedTupleType : ITypingTupleType { IReadOnlyList ItemNames { get; } + /// + /// Allows setting alternative name to the tuple at the variable assignment time. + /// + /// + /// Named tuple may get assigned to variables that have name different from the tuple itself. + /// Then the name may conflict with other types in module or its persistent model. For example, + /// 'tokenize' stub declares _TokenInfo = NamedTuple('TokenInfo', ...) but there is also + /// 'class TokenInfo(_TokenInfo)'' so we have to use the variable name in order to avoid type conflicts. + /// + /// + void SetName(string name); } } diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/AnyType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/AnyType.cs index f63241b4c..09881d8c7 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/AnyType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/AnyType.cs @@ -33,7 +33,7 @@ public AnyType(IPythonModule declaringModule) : base(declaringModule) { } public IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) => DeclaringModule.Interpreter.UnknownType; - public IPythonInstance CreateInstance(IArgumentSet args) => new PythonInstance(this); + public IMember CreateInstance(IArgumentSet args) => new PythonInstance(this); public IMember GetMember(string name) => null; public IEnumerable GetMemberNames() => Array.Empty(); diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericType.cs index 6bb6b2544..6133b0aa6 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericType.cs @@ -122,7 +122,7 @@ private SpecializedGenericType(string name, string qualifiedName, IPythonModule public bool IsAbstract => true; public bool IsSpecialized => true; - public IPythonInstance CreateInstance(IArgumentSet args) { + public IMember CreateInstance(IArgumentSet args) { var types = GetTypesFromValues(args.Arguments); if (types.Count != args.Arguments.Count) { throw new ArgumentException(@"Generic type instance construction arguments must be all of IPythonType", nameof(args)); diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericTypeParameter.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericTypeParameter.cs index be045b33a..3f430e1c2 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericTypeParameter.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericTypeParameter.cs @@ -71,7 +71,7 @@ private static bool TypeVarArgumentsValid(IArgumentSet argSet) { } // Report diagnostic if user passed in a value for name and it is not a string - var name = (args[0].Value as IPythonConstant)?.GetString(); + var name = (args[1].Value as IPythonConstant)?.GetString(); if (string.IsNullOrEmpty(name)) { eval.ReportDiagnostics( eval.Module.Uri, diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/NamedTupleType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/NamedTupleType.cs index 7e71f52e0..f568fd9fe 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/NamedTupleType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/NamedTupleType.cs @@ -25,6 +25,8 @@ namespace Microsoft.Python.Analysis.Specializations.Typing.Types { internal sealed class NamedTupleType : TypingTupleType, ITypingNamedTupleType { + private string _name; + // Since named tuple operates as a new, separate type, we need to track // its location rather than delegating down to the general wrapper over // Python built-in tuple. @@ -39,7 +41,7 @@ public NamedTupleLocatedMember(Location location) : base(location) { } /// public NamedTupleType(string tupleName, IReadOnlyList itemNames, IReadOnlyList itemTypes, IPythonModule declaringModule, IndexSpan indexSpan) : base(itemTypes, declaringModule, declaringModule.Interpreter) { - Name = tupleName ?? throw new ArgumentNullException(nameof(tupleName)); + _name = tupleName ?? throw new ArgumentNullException(nameof(tupleName)); ItemNames = itemNames; var typeNames = itemTypes.Select(t => t.IsUnknown() ? string.Empty : t.Name); @@ -49,10 +51,13 @@ public NamedTupleType(string tupleName, IReadOnlyList itemNames, IReadOn _locatedMember = new NamedTupleLocatedMember(new Location(declaringModule, indexSpan)); } + #region ITypingNamedTupleType public IReadOnlyList ItemNames { get; } + public void SetName(string name) => _name = name; + #endregion #region IPythonType - public override string Name { get; } + public override string Name => _name; public override string QualifiedName => $"{DeclaringModule.Name}:{Name}"; // Named tuple name is a type name as class. public override bool IsSpecialized => true; public override string Documentation { get; } @@ -66,7 +71,7 @@ public NamedTupleType(string tupleName, IReadOnlyList itemNames, IReadOn public override void RemoveReferences(IPythonModule module) => _locatedMember.RemoveReferences(module); #endregion - public override IPythonInstance CreateInstance(IArgumentSet args) => new TypingTuple(this); + public override IMember CreateInstance(IArgumentSet args) => new TypingTuple(this); // NamedTuple does not create instances, it defines a type. public override IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) => this; diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/OptionalType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/OptionalType.cs index 62cf914ad..79a948d8d 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/OptionalType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/OptionalType.cs @@ -31,7 +31,7 @@ public OptionalType(IPythonModule declaringModule, IPythonType type) : base(type public override bool IsSpecialized => true; public IEnumerator GetEnumerator() - => Enumerable.Repeat(DeclaringModule.Interpreter.GetBuiltinType(BuiltinTypeId.NoneType), 1) + => Enumerable.Repeat(DeclaringModule.Interpreter.GetBuiltinType(BuiltinTypeId.None), 1) .Concat(Enumerable.Repeat(InnerType, 1)).GetEnumerator(); IEnumerator IEnumerable.GetEnumerator() => GetEnumerator(); @@ -39,7 +39,7 @@ public IEnumerator GetEnumerator() public IPythonUnionType Add(IPythonType t) => this; public IPythonUnionType Add(IPythonUnionType types) => this; - public override IPythonInstance CreateInstance(IArgumentSet args) + public override IMember CreateInstance(IArgumentSet args) => InnerType.CreateInstance(args); } } diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypeAlias.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypeAlias.cs index 9c226e3af..8fbe478cd 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypeAlias.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypeAlias.cs @@ -25,6 +25,6 @@ public TypeAlias(string name, IPythonType type) : base(type) { public override string QualifiedName => $"typing:{Name}"; public override bool IsSpecialized => true; - public override IPythonInstance CreateInstance(IArgumentSet args) => new PythonInstance(this); + public override IMember CreateInstance(IArgumentSet args) => new PythonInstance(this); } } diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingDictionaryType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingDictionaryType.cs index de00a54fc..06a273895 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingDictionaryType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingDictionaryType.cs @@ -48,7 +48,7 @@ public TypingDictionaryType(string name, IPythonType keyType, IPythonType valueT public override string Name { get; } public override string QualifiedName { get; } - public override IPythonInstance CreateInstance(IArgumentSet args) => new TypingDictionary(this); + public override IMember CreateInstance(IArgumentSet args) => new TypingDictionary(this); public override IMember Index(IPythonInstance instance, IArgumentSet args) => ValueType.CreateInstance(args); public override bool IsSpecialized => true; diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingListType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingListType.cs index 7fd3a8234..88a4ffb8d 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingListType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingListType.cs @@ -52,7 +52,7 @@ public TypingListType(string typeName, BuiltinTypeId typeId, IPythonType itemTyp public override bool IsAbstract => false; public override bool IsSpecialized => true; - public override IPythonInstance CreateInstance(IArgumentSet args) => new TypingList(this); + public override IMember CreateInstance(IArgumentSet args) => new TypingList(this); public IPythonType ItemType { get; } public override IMember Index(IPythonInstance instance, IArgumentSet args) => ItemType.CreateInstance(args); diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingTupleType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingTupleType.cs index 8860aee39..caf77458d 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingTupleType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Types/TypingTupleType.cs @@ -45,7 +45,7 @@ public TypingTupleType(IReadOnlyList itemTypes, IPythonModule decla public override bool IsAbstract => false; public override bool IsSpecialized => true; - public override IPythonInstance CreateInstance(IArgumentSet args) + public override IMember CreateInstance(IArgumentSet args) => new TypingTuple(this); public override IMember Index(IPythonInstance instance, IArgumentSet args) { diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/TypingModule.cs b/src/Analysis/Ast/Impl/Specializations/Typing/TypingModule.cs index 54258b312..c02f4de8b 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/TypingModule.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/TypingModule.cs @@ -18,12 +18,12 @@ using Microsoft.Python.Analysis.Diagnostics; using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Specializations.Typing.Types; +using Microsoft.Python.Analysis.Specializations.Typing.Values; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Utilities; using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core; using Microsoft.Python.Core.Diagnostics; -using Microsoft.Python.Core.Text; using Microsoft.Python.Parsing; using Microsoft.Python.Parsing.Ast; @@ -49,39 +49,9 @@ public static IPythonModule Create(IServiceContainer services) { private void SpecializeMembers() { var location = new Location(this); - // TypeVar - var fn = PythonFunctionType.Specialize("TypeVar", this, GetMemberDocumentation("TypeVar")); - var o = new PythonFunctionOverload(fn, location); - o.SetParameters(new List { - new ParameterInfo("name", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, null), - new ParameterInfo("constraints", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.List, null), - new ParameterInfo("bound", Interpreter.GetBuiltinType(BuiltinTypeId.Type), ParameterKind.KeywordOnly, new PythonConstant(null, Interpreter.GetBuiltinType(BuiltinTypeId.NoneType))), - new ParameterInfo("covariant", Interpreter.GetBuiltinType(BuiltinTypeId.Bool), ParameterKind.KeywordOnly, new PythonConstant(false, Interpreter.GetBuiltinType(BuiltinTypeId.Bool))), - new ParameterInfo("contravariant", Interpreter.GetBuiltinType(BuiltinTypeId.Bool), ParameterKind.KeywordOnly, new PythonConstant(false, Interpreter.GetBuiltinType(BuiltinTypeId.Bool))) - }); - - // When called, create generic parameter type. For documentation - // use original TypeVar declaration so it appear as a tooltip. - o.SetReturnValueProvider((declaringModule, overload, args, indexSpan) - => GenericTypeParameter.FromTypeVar(args, declaringModule, indexSpan)); - - fn.AddOverload(o); - _members["TypeVar"] = fn; - - // NewType + _members["TypeVar"] = new TypeVar(this); _members["NewType"] = SpecializeNewType(location); - - // Type - fn = PythonFunctionType.Specialize("Type", this, GetMemberDocumentation("Type")); - o = new PythonFunctionOverload(fn, location); - // When called, create generic parameter type. For documentation - // use original TypeVar declaration so it appear as a tooltip. - o.SetReturnValueProvider((declaringModule, overload, args, indexSpan) => { - var a = args.Values(); - return a.Count == 1 ? a[0] : Interpreter.UnknownType; - }); - fn.AddOverload(o); - _members["Type"] = fn; + _members["Type"] = new Type(this); _members["Iterator"] = new SpecializedGenericType("Iterator", CreateIteratorType, this); @@ -133,12 +103,7 @@ private void SpecializeMembers() { //_members["SupportsBytes"] = Interpreter.GetBuiltinType(BuiltinTypeId.Bytes); _members["ByteString"] = Interpreter.GetBuiltinType(BuiltinTypeId.Bytes); - fn = PythonFunctionType.Specialize("NamedTuple", this, GetMemberDocumentation("NamedTuple")); - o = new PythonFunctionOverload(fn, location); - o.SetReturnValueProvider((declaringModule, overload, args, indexSpan) - => CreateNamedTuple(args.Values(), declaringModule, indexSpan)); - fn.AddOverload(o); - _members["NamedTuple"] = fn; + _members["NamedTuple"] = new NamedTuple(this); _members["Any"] = new AnyType(this); _members["AnyStr"] = CreateAnyStr(); @@ -282,54 +247,6 @@ private IPythonType CreateUnion(IReadOnlyList typeArgs) { return Interpreter.UnknownType; } - private IPythonType CreateNamedTuple(IReadOnlyList typeArgs, IPythonModule declaringModule, IndexSpan indexSpan) { - if (typeArgs.Count != 2) { - // TODO: report wrong number of arguments - return Interpreter.UnknownType; - } - - ; - if (!typeArgs[0].TryGetConstant(out var tupleName) || string.IsNullOrEmpty(tupleName)) { - // TODO: report name is incorrect. - return Interpreter.UnknownType; - } - - var argList = (typeArgs[1] as IPythonCollection)?.Contents; - if (argList == null) { - // TODO: report type spec is not a list. - return Interpreter.UnknownType; - } - - var itemNames = new List(); - var itemTypes = new List(); - foreach (var a in argList) { - if (a.TryGetConstant(out string itemName1)) { - // Not annotated - itemNames.Add(itemName1); - itemTypes.Add(Interpreter.UnknownType); - continue; - } - - // Now assume annotated pair that comes as a tuple. - if (!(a is IPythonCollection c) || c.Type.TypeId != BuiltinTypeId.Tuple) { - // TODO: report that item is not a tuple. - continue; - } - if (c.Contents.Count != 2) { - // TODO: report extra items in the element spec. - continue; - } - if (!c.Contents[0].TryGetConstant(out var itemName2)) { - // TODO: report item name is not a string. - continue; - } - - itemNames.Add(itemName2); - itemTypes.Add(c.Contents[1].GetPythonType()); - } - return TypingTypeFactory.CreateNamedTupleType(tupleName, itemNames, itemTypes, declaringModule, indexSpan); - } - private IPythonType CreateOptional(IReadOnlyList typeArgs) { if (typeArgs.Count == 1) { return typeArgs[0]; diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Values/NamedTuple.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Values/NamedTuple.cs new file mode 100644 index 000000000..cc6b1d53b --- /dev/null +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Values/NamedTuple.cs @@ -0,0 +1,99 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core.Text; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Specializations.Typing.Values { + /// + /// Specialization of NamedTuple(). + /// + internal sealed class NamedTuple : SpecializedClass { + private readonly IPythonFunctionType _constructor; + + public NamedTuple(IPythonModule declaringModule) : base(BuiltinTypeId.Tuple, declaringModule) { + var interpreter = DeclaringModule.Interpreter; + + var fn = new PythonFunctionType("__init__", new Location(declaringModule), this, "NamedTuple"); + var o = new PythonFunctionOverload(fn, new Location(DeclaringModule)); + + o.SetParameters(new List { + new ParameterInfo("self", this, ParameterKind.Normal, this), + new ParameterInfo("name", interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, null), + new ParameterInfo("members", interpreter.GetBuiltinType(BuiltinTypeId.List), ParameterKind.Normal, null) + }); + fn.AddOverload(o); + _constructor = fn; + } + + public override IMember GetMember(string name) + => name == _constructor.Name ? _constructor : base.GetMember(name); + + public override IMember CreateInstance(IArgumentSet args) + => CreateNamedTuple(args.Values(), DeclaringModule, default); + + private IPythonType CreateNamedTuple(IReadOnlyList typeArgs, IPythonModule declaringModule, IndexSpan indexSpan) { + // For class argument list includes 'self' + if (typeArgs.Count != 3) { + // TODO: report wrong number of arguments + return DeclaringModule.Interpreter.UnknownType; + } + + ; + if (!typeArgs[1].TryGetConstant(out var tupleName) || string.IsNullOrEmpty(tupleName)) { + // TODO: report name is incorrect. + return DeclaringModule.Interpreter.UnknownType; + } + + var argList = (typeArgs[2] as IPythonCollection)?.Contents; + if (argList == null) { + // TODO: report type spec is not a list. + return DeclaringModule.Interpreter.UnknownType; + } + + var itemNames = new List(); + var itemTypes = new List(); + foreach (var a in argList) { + if (a.TryGetConstant(out string itemName1)) { + // Not annotated + itemNames.Add(itemName1); + itemTypes.Add(DeclaringModule.Interpreter.UnknownType); + continue; + } + + // Now assume annotated pair that comes as a tuple. + if (!(a is IPythonCollection c) || c.Type.TypeId != BuiltinTypeId.Tuple) { + // TODO: report that item is not a tuple. + continue; + } + if (c.Contents.Count != 2) { + // TODO: report extra items in the element spec. + continue; + } + if (!c.Contents[0].TryGetConstant(out var itemName2)) { + // TODO: report item name is not a string. + continue; + } + + itemNames.Add(itemName2); + itemTypes.Add(c.Contents[1].GetPythonType()); + } + return TypingTypeFactory.CreateNamedTupleType(tupleName, itemNames, itemTypes, declaringModule, indexSpan); + } + } +} diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Values/SpecializedClass.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Values/SpecializedClass.cs new file mode 100644 index 000000000..933c0e081 --- /dev/null +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Values/SpecializedClass.cs @@ -0,0 +1,43 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Specializations.Typing.Values { + internal abstract class SpecializedClass: PythonTypeWrapper, IPythonClassType { + protected SpecializedClass(BuiltinTypeId typeId, IPythonModule declaringModule) + : base(typeId, declaringModule) { + } + + #region IPythonClassType + public override PythonMemberType MemberType => PythonMemberType.Class; + public override IMember Call(IPythonInstance instance, string memberName, IArgumentSet argSet) => this; + public override IMember Index(IPythonInstance instance, IArgumentSet args) => null; + public IPythonType CreateSpecificType(IArgumentSet typeArguments) => this; + public IPythonType DeclaringType => null; + public IReadOnlyList Parameters => Array.Empty(); + public bool IsGeneric => false; + public ClassDefinition ClassDefinition => null; + public IReadOnlyList Mro => Array.Empty(); + public IReadOnlyList Bases => new[] { DeclaringModule.Interpreter.GetBuiltinType(BuiltinTypeId.Object) }; + public IReadOnlyDictionary GenericParameters => EmptyDictionary.Instance; + #endregion + } +} diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Values/Type.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Values/Type.cs new file mode 100644 index 000000000..0b8323fce --- /dev/null +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Values/Type.cs @@ -0,0 +1,31 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using Microsoft.Python.Analysis.Specializations.Typing.Values; +using Microsoft.Python.Analysis.Types; + +namespace Microsoft.Python.Analysis.Specializations { + internal sealed class Type : SpecializedClass { + public Type(IPythonModule declaringModule) : base(BuiltinTypeId.Type, declaringModule) { } + + public override PythonMemberType MemberType => PythonMemberType.Class; + + public override IMember CreateInstance(IArgumentSet args) { + var argMembers = args.Values(); + // type(self, ...) + return argMembers.Count > 1 ? argMembers[1].GetPythonType().ToBound() : this; + } + } +} diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypeVar.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypeVar.cs new file mode 100644 index 000000000..35a451743 --- /dev/null +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypeVar.cs @@ -0,0 +1,54 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using Microsoft.Python.Analysis.Specializations.Typing.Types; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Specializations.Typing.Values { + /// + /// Specialization of TypeVar(). + /// + internal sealed class TypeVar : SpecializedClass { + private readonly IPythonFunctionType _constructor; + + public TypeVar(IPythonModule declaringModule) : base(BuiltinTypeId.Type, declaringModule) { + var interpreter = DeclaringModule.Interpreter; + + var fn = new PythonFunctionType("__init__", new Location(declaringModule), this, "TypeVar"); + var o = new PythonFunctionOverload(fn, new Location(DeclaringModule)); + + var boolType = interpreter.GetBuiltinType(BuiltinTypeId.Bool); + o.SetParameters(new List { + new ParameterInfo("self", this, ParameterKind.Normal, this), + new ParameterInfo("name", interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.Normal, null), + new ParameterInfo("constraints", interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.List, null), + new ParameterInfo("bound", interpreter.GetBuiltinType(BuiltinTypeId.Type), ParameterKind.KeywordOnly, new PythonConstant(null, interpreter.GetBuiltinType(BuiltinTypeId.None))), + new ParameterInfo("covariant", boolType, ParameterKind.KeywordOnly, new PythonConstant(false, boolType)), + new ParameterInfo("contravariant", boolType, ParameterKind.KeywordOnly, new PythonConstant(false, boolType)) + }); + fn.AddOverload(o); + _constructor = fn; + } + + public override IMember GetMember(string name) + => name == _constructor.Name ? _constructor : base.GetMember(name); + + public override IMember CreateInstance(IArgumentSet args) + => GenericTypeParameter.FromTypeVar(args, args.Eval?.Module ?? DeclaringModule); + } +} diff --git a/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypingType.cs b/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypingType.cs index e05fbeea1..c12d76f09 100644 --- a/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypingType.cs +++ b/src/Analysis/Ast/Impl/Specializations/Typing/Values/TypingType.cs @@ -41,7 +41,7 @@ public TypingType(IPythonModule declaringModule, IPythonType type): base(declari public bool IsSpecialized => true; public IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) => _type.Call(instance, memberName, args); - public IPythonInstance CreateInstance(IArgumentSet args) => _type.CreateInstance(args); + public IMember CreateInstance(IArgumentSet args) => _type.CreateInstance(args); public IMember GetMember(string name) => _type.GetMember(name); public IEnumerable GetMemberNames() => _type.GetMemberNames(); public IMember Index(IPythonInstance instance, IArgumentSet args) => _type.Index(instance, args); diff --git a/src/Analysis/Ast/Impl/Types/ArgumentSet.cs b/src/Analysis/Ast/Impl/Types/ArgumentSet.cs index 95d7d91e2..efd16cd3c 100644 --- a/src/Analysis/Ast/Impl/Types/ArgumentSet.cs +++ b/src/Analysis/Ast/Impl/Types/ArgumentSet.cs @@ -15,6 +15,7 @@ // permissions and limitations under the License. using System.Collections.Generic; +using System.Diagnostics; using System.IO; using System.Linq; using Microsoft.Python.Analysis.Analyzer; @@ -100,7 +101,7 @@ public ArgumentSet(IPythonFunctionType fn, int overloadIndex, IPythonType instan } var overload = fn.Overloads[overloadIndex]; - var fd = overload.FunctionDefinition; + var fdParameters = overload.FunctionDefinition?.Parameters.Where(p => !p.IsPositionalOnlyMarker).ToArray(); // Some specialized functions have more complicated definitions, so we pass // parameters to those, TypeVar() is an example, so we allow the latter logic to handle @@ -115,7 +116,7 @@ public ArgumentSet(IPythonFunctionType fn, int overloadIndex, IPythonType instan if (string.IsNullOrEmpty(name)) { name = i < overload.Parameters.Count ? overload.Parameters[i].Name : $"arg{i}"; } - var node = fd != null && i < fd.Parameters.Length ? fd.Parameters[i] : null; + var node = fdParameters?.ElementAtOrDefault(i); _arguments.Add(new Argument(name, ParameterKind.Normal, callExpr.Args[i].Expression, null, node)); } return; @@ -131,7 +132,7 @@ public ArgumentSet(IPythonFunctionType fn, int overloadIndex, IPythonType instan var slots = new Argument[overload.Parameters.Count]; for (var i = 0; i < overload.Parameters.Count; i++) { - var node = fd != null && i < fd.Parameters.Length ? fd.Parameters[i] : null; + var node = fdParameters?.ElementAtOrDefault(i); slots[i] = new Argument(overload.Parameters[i], node); } @@ -250,6 +251,12 @@ public ArgumentSet(IPythonFunctionType fn, int overloadIndex, IPythonType instan continue; } + if (nvp.Kind == ParameterKind.PositionalOnly) { + _errors.Add(new DiagnosticsEntry(Resources.Analysis_PositionalOnlyArgumentNamed.FormatInvariant(arg.Name), arg.GetLocation(eval).Span, + ErrorCodes.PositionalOnlyNamed, Severity.Warning, DiagnosticSource.Analysis)); + return; + } + if (nvp.ValueExpression != null || nvp.Value != null) { // Slot is already filled. _errors.Add(new DiagnosticsEntry(Resources.Analysis_ParameterAlreadySpecified.FormatUI(arg.Name), arg.GetLocation(eval).Span, @@ -286,25 +293,25 @@ public ArgumentSet(IPythonFunctionType fn, int overloadIndex, IPythonType instan } } - public ArgumentSet Evaluate() { + public ArgumentSet Evaluate(LookupOptions lookupOptions = LookupOptions.Normal) { if (_evaluated || Eval == null) { return this; } foreach (var a in _arguments.Where(x => x.Value == null)) { - a.Value = GetArgumentValue(a); + a.Value = GetArgumentValue(a, lookupOptions); } if (_listArgument != null) { foreach (var e in _listArgument.Expressions) { - var value = Eval.GetValueFromExpression(e) ?? Eval.UnknownType; + var value = Eval.GetValueFromExpression(e, lookupOptions) ?? Eval.UnknownType; _listArgument._Values.Add(value); } } if (_dictArgument != null) { foreach (var e in _dictArgument.Expressions) { - var value = Eval.GetValueFromExpression(e.Value) ?? Eval.UnknownType; + var value = Eval.GetValueFromExpression(e.Value, lookupOptions) ?? Eval.UnknownType; _dictArgument._Args[e.Key] = value; } } @@ -313,7 +320,7 @@ public ArgumentSet Evaluate() { return this; } - private IMember GetArgumentValue(Argument arg) { + private IMember GetArgumentValue(Argument arg, LookupOptions lookupOptions) { if (arg.Value is IMember m) { return m; } @@ -326,10 +333,10 @@ private IMember GetArgumentValue(Argument arg) { if (arg.ValueIsDefault) { using (Eval.OpenScope(DeclaringModule.GlobalScope)) { - return Eval.GetValueFromExpression(arg.ValueExpression) ?? Eval.UnknownType; + return Eval.GetValueFromExpression(arg.ValueExpression, lookupOptions) ?? Eval.UnknownType; } } - return Eval.GetValueFromExpression(arg.ValueExpression) ?? Eval.UnknownType; + return Eval.GetValueFromExpression(arg.ValueExpression, lookupOptions) ?? Eval.UnknownType; } private Expression CreateExpression(string paramName, string defaultValue) { @@ -346,6 +353,7 @@ private Expression CreateExpression(string paramName, string defaultValue) { } } + [DebuggerDisplay("{Name} : {Kind}")] private sealed class Argument : IArgument { /// /// Argument name. diff --git a/src/Analysis/Ast/Impl/Types/Collections/PythonCollectionType.cs b/src/Analysis/Ast/Impl/Types/Collections/PythonCollectionType.cs index 7e8c16fc2..64ac83c60 100644 --- a/src/Analysis/Ast/Impl/Types/Collections/PythonCollectionType.cs +++ b/src/Analysis/Ast/Impl/Types/Collections/PythonCollectionType.cs @@ -57,7 +57,7 @@ bool isMutable public override PythonMemberType MemberType => PythonMemberType.Class; public override IMember GetMember(string name) => name == @"__iter__" ? IteratorType : base.GetMember(name); - public override IPythonInstance CreateInstance(IArgumentSet args) + public override IMember CreateInstance(IArgumentSet args) => new PythonCollection(this, args.Arguments.Select(a => a.Value).OfType().ToArray()); public override IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) @@ -67,7 +67,7 @@ public override IMember Index(IPythonInstance instance, IArgumentSet args) => (instance as IPythonCollection)?.Index(args) ?? UnknownType; public IPythonType CreateSpecificType(IArgumentSet typeArguments) { - throw new NotImplementedException(); + return this; } #endregion diff --git a/src/Analysis/Ast/Impl/Types/Collections/PythonDictionaryType.cs b/src/Analysis/Ast/Impl/Types/Collections/PythonDictionaryType.cs index d528a8d21..29d4c619e 100644 --- a/src/Analysis/Ast/Impl/Types/Collections/PythonDictionaryType.cs +++ b/src/Analysis/Ast/Impl/Types/Collections/PythonDictionaryType.cs @@ -24,7 +24,7 @@ public PythonDictionaryType(IPythonModule declaringModule, bool isMutable = true : base(BuiltinTypeId.Dict, declaringModule, isMutable) { } - public override IPythonInstance CreateInstance(IArgumentSet args) { + public override IMember CreateInstance(IArgumentSet args) { var contents = args.Arguments.Count == 1 ? args.Arguments[0].Value as IReadOnlyDictionary : EmptyDictionary.Instance; diff --git a/src/Analysis/Ast/Impl/Types/Definitions/BuiltinTypeId.cs b/src/Analysis/Ast/Impl/Types/Definitions/BuiltinTypeId.cs index e7dffcf4d..e914ab967 100644 --- a/src/Analysis/Ast/Impl/Types/Definitions/BuiltinTypeId.cs +++ b/src/Analysis/Ast/Impl/Types/Definitions/BuiltinTypeId.cs @@ -21,7 +21,7 @@ public enum BuiltinTypeId { Unknown, Object, Type, - NoneType, + None, Bool, Int, diff --git a/src/Analysis/Ast/Impl/Types/Definitions/IBuiltinPythonModule.cs b/src/Analysis/Ast/Impl/Types/Definitions/IBuiltinPythonModule.cs index 28f4d1177..03ed6a9c4 100644 --- a/src/Analysis/Ast/Impl/Types/Definitions/IBuiltinPythonModule.cs +++ b/src/Analysis/Ast/Impl/Types/Definitions/IBuiltinPythonModule.cs @@ -17,12 +17,11 @@ namespace Microsoft.Python.Analysis.Types { /// /// Represents a built-in Python module. The built-in module needs to respond to /// some extra requests for members by name which supports getting hidden members - /// such as "NoneType" which logically live in the built-in module but don't actually + /// which logically live in the built-in module but don't actually /// exist there by name. /// /// The full list of types which will be accessed through GetAnyMember but don't exist /// in the built-in module includes: - /// NoneType /// generator /// builtin_function /// builtin_method_descriptor diff --git a/src/Analysis/Ast/Impl/Types/Definitions/IPythonModule.cs b/src/Analysis/Ast/Impl/Types/Definitions/IPythonModule.cs index 168670be1..9b438e082 100644 --- a/src/Analysis/Ast/Impl/Types/Definitions/IPythonModule.cs +++ b/src/Analysis/Ast/Impl/Types/Definitions/IPythonModule.cs @@ -56,7 +56,7 @@ public interface IPythonModule : IPythonType { /// Global cope of the module. /// IGlobalScope GlobalScope { get; } - + /// /// If module is a stub points to the primary module. /// Typically used in code navigation scenarios when user @@ -68,5 +68,13 @@ public interface IPythonModule : IPythonType { /// Indicates if module is restored from database. /// bool IsPersistent { get; } + + /// + /// Defines if module belongs to Typeshed and hence resolved + /// via typeshed module resolution service. + /// + bool IsTypeshed { get; } + + ModuleState ModuleState { get; } } } diff --git a/src/Analysis/Ast/Impl/Types/Definitions/IPythonSuperType.cs b/src/Analysis/Ast/Impl/Types/Definitions/IPythonSuperType.cs new file mode 100644 index 000000000..b2dcc58a8 --- /dev/null +++ b/src/Analysis/Ast/Impl/Types/Definitions/IPythonSuperType.cs @@ -0,0 +1,28 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; + +namespace Microsoft.Python.Analysis.Types { + /// + /// Represents Python class type definition. + /// + public interface IPythonSuperType { + /// + /// Python Method Resolution Order (MRO). + /// + IReadOnlyList Mro { get; } + } +} diff --git a/src/Analysis/Ast/Impl/Types/Definitions/IPythonType.cs b/src/Analysis/Ast/Impl/Types/Definitions/IPythonType.cs index d546f7261..77a45452e 100644 --- a/src/Analysis/Ast/Impl/Types/Definitions/IPythonType.cs +++ b/src/Analysis/Ast/Impl/Types/Definitions/IPythonType.cs @@ -59,24 +59,22 @@ public interface IPythonType : ILocatedMember, IMemberContainer { /// /// Create instance of the type, if any. /// - /// Name of the type. Used in specialization scenarios - /// where constructor may want to create specialized type. - /// Any custom arguments required to create the instance. - IPythonInstance CreateInstance(IArgumentSet args); + /// Call arguments. + IMember CreateInstance(IArgumentSet args); /// /// Invokes method or property on the specified instance. /// /// Instance of the type. /// Member name to call, if applicable. - /// Call arguments. - IMember Call(IPythonInstance instance, string memberName, IArgumentSet argSet); + /// Call arguments. + IMember Call(IPythonInstance instance, string memberName, IArgumentSet args); /// /// Invokes indexer on the specified instance. /// /// Instance of the type. - /// Index arguments. + /// Call arguments. IMember Index(IPythonInstance instance, IArgumentSet args); } } diff --git a/src/Analysis/Ast/Impl/Types/LocatedMember.cs b/src/Analysis/Ast/Impl/Types/LocatedMember.cs index 2a70b4004..288a9fe35 100644 --- a/src/Analysis/Ast/Impl/Types/LocatedMember.cs +++ b/src/Analysis/Ast/Impl/Types/LocatedMember.cs @@ -18,12 +18,16 @@ using System.Linq; using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Core; +using Microsoft.Python.Core.Diagnostics; namespace Microsoft.Python.Analysis.Types { internal abstract class LocatedMember : ILocatedMember { private HashSet _references; - protected LocatedMember(IPythonModule module) : this(new Location(module)) { } + protected LocatedMember(IPythonModule module) : this(new Location(module)) { + Check.InvalidOperation(module != null || this is IPythonModule, + "Located member can only have null declaring module if it is the module."); + } protected LocatedMember(Location location) { Location = location; @@ -54,11 +58,17 @@ public virtual IReadOnlyList References { public virtual void AddReference(Location location) { lock (this) { - if(this.DeclaringModule == null || this.DeclaringModule?.ModuleType == ModuleType.Builtins) { + // In order to limit memory consumption we normally don't track references + // to builtin types such as int or list. Exception is functions like 'print' + // since it user may want to find all references to them. + if (this.DeclaringModule == null || + (this.DeclaringModule?.ModuleType == ModuleType.Builtins && MemberType != PythonMemberType.Function)) { return; } + var keepReferencesInLibraries = + location.Module.Analysis.ExpressionEvaluator.Services.GetService()?.Options.KeepLibraryAst == true; // Don't add references to library code. - if (location.Module?.ModuleType == ModuleType.User && !location.Equals(Location)) { + if ((location.Module?.ModuleType == ModuleType.User || keepReferencesInLibraries) && !location.Equals(Location)) { _references = _references ?? new HashSet(); _references.Add(location); } diff --git a/src/Analysis/Ast/Impl/Types/Location.cs b/src/Analysis/Ast/Impl/Types/Location.cs index e7f49e0bd..53060d859 100644 --- a/src/Analysis/Ast/Impl/Types/Location.cs +++ b/src/Analysis/Ast/Impl/Types/Location.cs @@ -18,9 +18,7 @@ namespace Microsoft.Python.Analysis.Types { public readonly struct Location { - public Location(IPythonModule module) : this(module, default) { } - - public Location(IPythonModule module, IndexSpan indexSpan) { + public Location(IPythonModule module, IndexSpan indexSpan = default) { Module = module; IndexSpan = indexSpan; } diff --git a/src/Analysis/Ast/Impl/Types/ParameterInfo.cs b/src/Analysis/Ast/Impl/Types/ParameterInfo.cs index c03e8b56a..12d464acf 100644 --- a/src/Analysis/Ast/Impl/Types/ParameterInfo.cs +++ b/src/Analysis/Ast/Impl/Types/ParameterInfo.cs @@ -13,10 +13,11 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. -using System; +using System.Diagnostics; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.Analysis.Types { + [DebuggerDisplay("{Name} : {Kind}")] internal sealed class ParameterInfo : IParameterInfo { public ParameterInfo(PythonAst ast, Parameter p, IPythonType type, IMember defaultValue, bool isGeneric) : this(p?.Name, type, p?.Kind, defaultValue) { @@ -34,6 +35,7 @@ public ParameterInfo(string name, IPythonType type, ParameterKind? kind, IMember DefaultValue = defaultValue; Type = type; Kind = kind ?? ParameterKind.Normal; + Debug.Assert(Kind != ParameterKind.PositionalOnlyMarker); } public string Name { get; } diff --git a/src/Analysis/Ast/Impl/Types/PythonClassType.cs b/src/Analysis/Ast/Impl/Types/PythonClassType.cs index 708f254d4..3ba89e2a0 100644 --- a/src/Analysis/Ast/Impl/Types/PythonClassType.cs +++ b/src/Analysis/Ast/Impl/Types/PythonClassType.cs @@ -79,10 +79,8 @@ public override IEnumerable GetMemberNames() { } public override IMember GetMember(string name) { - IMember member; - lock (_membersLock) { - if (Members.TryGetValue(name, out member)) { + if (Members.TryGetValue(name, out var member)) { return member; } } @@ -101,12 +99,7 @@ public override IMember GetMember(string name) { using (_memberGuard.Push(this, out var reentered)) { if (!reentered) { - foreach (var m in Mro.Reverse()) { - if (m == this) { - return member; - } - member = member ?? m.GetMember(name); - } + return Mro.Skip(1).Select(c => c.GetMember(name)).ExcludeDefault().FirstOrDefault(); } return null; } @@ -149,7 +142,7 @@ public override string Documentation { } // Constructor call - public override IPythonInstance CreateInstance(IArgumentSet args) { + public override IMember CreateInstance(IArgumentSet args) { var builtins = DeclaringModule.Interpreter.ModuleResolution.BuiltinsModule; // Specializations switch (Name) { @@ -165,6 +158,12 @@ public override IPythonInstance CreateInstance(IArgumentSet args) { return PythonCollectionType.CreateTuple(builtins, contents); } } + + // Metaclasses return type, not instance. + if (Bases.MaybeEnumerate().Any(b => b.Name == "type" && b.DeclaringModule.ModuleType == ModuleType.Builtins)) { + return this; + } + return new PythonInstance(this); } @@ -188,7 +187,7 @@ public override IMember Index(IPythonInstance instance, IArgumentSet args) { public ClassDefinition ClassDefinition => DeclaringModule.GetAstNode(this); public IReadOnlyList Bases { get { - lock(_membersLock) { + lock (_membersLock) { return _bases?.ToArray(); } } @@ -348,7 +347,7 @@ private IEnumerable DisambiguateBases(IEnumerable base // Get locally declared variable, make sure it is a declaration // and that it declared a class. var lv = scope.Variables[b.Name]; - if (lv.Source != VariableSource.Import && lv.Value is IPythonClassType cls && cls.IsDeclaredAfterOrAt(this.Location)) { + if (lv.Source != VariableSource.Import && lv.Value is IPythonClassType cls && cls.IsDeclaredAfterOrAt(Location)) { // There is a declaration with the same name, but it appears later in the module. Use the import. if (!importedType.IsUnknown()) { newBases.Add(importedType); diff --git a/src/Analysis/Ast/Impl/Types/PythonFunctionType.cs b/src/Analysis/Ast/Impl/Types/PythonFunctionType.cs index e28025c76..1d9b20eb6 100644 --- a/src/Analysis/Ast/Impl/Types/PythonFunctionType.cs +++ b/src/Analysis/Ast/Impl/Types/PythonFunctionType.cs @@ -20,7 +20,6 @@ using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core; using Microsoft.Python.Core.Collections; -using Microsoft.Python.Core.Diagnostics; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.Analysis.Types { diff --git a/src/Analysis/Ast/Impl/Types/PythonSuperType.cs b/src/Analysis/Ast/Impl/Types/PythonSuperType.cs new file mode 100644 index 000000000..8511c6173 --- /dev/null +++ b/src/Analysis/Ast/Impl/Types/PythonSuperType.cs @@ -0,0 +1,75 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Core; + +namespace Microsoft.Python.Analysis.Types { + internal sealed class PythonSuperType : PythonType, IPythonSuperType { + /// + /// Implements 'super' specialization type. + /// See also https://docs.python.org/3/library/functions.html#super + /// + /// The derived class MRO. + /// Builtins module. + public PythonSuperType(IReadOnlyList mro, IBuiltinsPythonModule builtins) + : base("super", new Location(builtins), string.Empty, BuiltinTypeId.Type) { + Mro = mro; + } + + public override string QualifiedName => $":SuperType[{string.Join(",", Mro.Select(t => t.QualifiedName))}]"; + + public IReadOnlyList Mro { get; } + + public override IMember GetMember(string name) + => Mro.MaybeEnumerate().Select(c => c.GetMember(name)).ExcludeDefault().FirstOrDefault(); + + public override IEnumerable GetMemberNames() + => Mro.MaybeEnumerate().SelectMany(cls => cls.GetMemberNames()).Distinct(); + + /// + /// Creates PythonSuperType with a MRO list that starts with the next class in line in classType's MRO, + /// or searches classType's MRO for , then builds an MRO list for + /// the remaining classes after . + /// + /// + /// + /// + internal static PythonSuperType Create(IPythonClassType classType, IPythonType typeToFind = null) { + var mro = classType?.Mro ?? Array.Empty(); + if (classType == null || mro.Count == 0) { + return null; + } + + // skip doing work if the newStartType is the first element in the callers mro + if (typeToFind?.Equals(classType.Mro.FirstOrDefault()) == false) { + var mroList = classType.Mro.ToList(); + var start = mroList.FindIndex(0, t => t.Equals(typeToFind)); + if (start >= 0) { + mro = mroList.GetRange(start, mro.Count - start).ToArray(); + } else { + return null; // typeToFind wasn't in the mro + } + } + + var nextClassInLine = mro?.FirstOrDefault(); + var builtins = classType.DeclaringModule.Interpreter.ModuleResolution.BuiltinsModule; + // Skip the first element, super's search starts at the next element in the mro for both super() and super(cls, typeToFind) + return nextClassInLine != null ? new PythonSuperType(mro.Skip(1).ToArray(), builtins) : null; + } + } +} diff --git a/src/Analysis/Ast/Impl/Types/PythonType.cs b/src/Analysis/Ast/Impl/Types/PythonType.cs index 4abb24c38..11770248a 100644 --- a/src/Analysis/Ast/Impl/Types/PythonType.cs +++ b/src/Analysis/Ast/Impl/Types/PythonType.cs @@ -47,14 +47,6 @@ private PythonType(string name, Location location, BuiltinTypeId typeId) : base( #region ILocatedMember public override PythonMemberType MemberType => _typeId.GetMemberId(); - - public override void AddReference(Location location) { - if (DeclaringModule == null || DeclaringModule.ModuleType == ModuleType.Builtins) { - return; - } - - base.AddReference(location); - } #endregion #region IPythonType @@ -75,10 +67,8 @@ public virtual string QualifiedName /// /// Create instance of the type, if any. /// - /// Name of the type. Used in specialization scenarios - /// where constructor may want to create specialized type. /// Any custom arguments required to create the instance. - public virtual IPythonInstance CreateInstance(IArgumentSet args) => new PythonInstance(this); + public virtual IMember CreateInstance(IArgumentSet args) => new PythonInstance(this); /// /// Invokes method or property on the specified instance. diff --git a/src/Analysis/Ast/Impl/Types/PythonTypeWrapper.cs b/src/Analysis/Ast/Impl/Types/PythonTypeWrapper.cs index 3c38194b1..5b129ac91 100644 --- a/src/Analysis/Ast/Impl/Types/PythonTypeWrapper.cs +++ b/src/Analysis/Ast/Impl/Types/PythonTypeWrapper.cs @@ -34,7 +34,7 @@ protected IPythonType InnerType /// Creates delegate type wrapper over an existing type. /// Use dedicated constructor for wrapping builtin types. /// - public PythonTypeWrapper(IPythonType type) : this(type, type.DeclaringModule) { } + protected PythonTypeWrapper(IPythonType type) : this(type, type.DeclaringModule) { } public PythonTypeWrapper(string typeName, string documentation, IPythonModule declaringModule, IPythonType baseType) : this(baseType, declaringModule) { _typeName = typeName; @@ -45,7 +45,7 @@ public PythonTypeWrapper(string typeName, string documentation, IPythonModule de /// Creates delegate type wrapper over an existing type. /// Use dedicated constructor for wrapping builtin types. /// - public PythonTypeWrapper(IPythonType type, IPythonModule declaringModule) { + protected PythonTypeWrapper(IPythonType type, IPythonModule declaringModule) { _innerType = type ?? throw new ArgumentNullException(nameof(type)); DeclaringModule = declaringModule; } @@ -55,7 +55,7 @@ public PythonTypeWrapper(IPythonType type, IPythonModule declaringModule) { /// wrap builtins since it can be done when builtins module is not loaded /// yet - such as when builtins module itself is being imported or specialized. /// - public PythonTypeWrapper(BuiltinTypeId builtinTypeId, IPythonModule declaringModule) { + protected PythonTypeWrapper(BuiltinTypeId builtinTypeId, IPythonModule declaringModule) { DeclaringModule = declaringModule ?? throw new ArgumentNullException(nameof(declaringModule)); _builtinTypeId = builtinTypeId; } @@ -71,7 +71,7 @@ public PythonTypeWrapper(BuiltinTypeId builtinTypeId, IPythonModule declaringMod public virtual bool IsAbstract => InnerType.IsAbstract; public virtual bool IsSpecialized => InnerType.IsSpecialized; - public virtual IPythonInstance CreateInstance(IArgumentSet args) + public virtual IMember CreateInstance(IArgumentSet args) => IsAbstract ? null : InnerType.CreateInstance(args); public virtual IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) => InnerType.Call(instance, memberName, args); diff --git a/src/Analysis/Ast/Impl/Types/PythonUnionType.cs b/src/Analysis/Ast/Impl/Types/PythonUnionType.cs index 6888d657a..eef3c019c 100644 --- a/src/Analysis/Ast/Impl/Types/PythonUnionType.cs +++ b/src/Analysis/Ast/Impl/Types/PythonUnionType.cs @@ -28,15 +28,19 @@ internal sealed class PythonUnionType : LocatedMember, IPythonUnionType { public PythonUnionType(IEnumerable types, IPythonModule declaringModule) : base(declaringModule.Interpreter.ModuleResolution.GetSpecializedModule("typing")) { - _types.UnionWith(types); + _types.UnionWith(types.Where(t => !this.Equals(t))); } private PythonUnionType(IPythonType x, IPythonType y) : base(x.DeclaringModule.Interpreter.ModuleResolution.GetSpecializedModule("typing")) { Check.Argument(nameof(x), () => !(x is IPythonUnionType)); Check.Argument(nameof(y), () => !(y is IPythonUnionType)); - _types.Add(x); - _types.Add(y); + if (!this.Equals(x)) { + _types.Add(x); + } + if (!this.Equals(y)) { + _types.Add(y); + } } public override PythonMemberType MemberType => PythonMemberType.Union; @@ -69,7 +73,7 @@ public bool IsBuiltin { public bool IsAbstract => false; public bool IsSpecialized => true; - public IPythonInstance CreateInstance(IArgumentSet args) => new PythonUnion(this); + public IMember CreateInstance(IArgumentSet args) => new PythonUnion(this); public IMember Call(IPythonInstance instance, string memberName, IArgumentSet args) { IPythonType[] types; @@ -91,7 +95,7 @@ public IMember Index(IPythonInstance instance, IArgumentSet args) { // Check if any types support indexing var result = types .Select(t => t.Index(instance, args)) - .FirstOrDefault(r => !r.IsUnknown() && r.GetPythonType() != this); + .FirstOrDefault(r => !r.IsUnknown() && !r.GetPythonType().Equals(this)); return result ?? DeclaringModule.Interpreter.UnknownType; } @@ -112,7 +116,7 @@ public IPythonUnionType Add(IPythonType t) { public IPythonUnionType Add(IPythonUnionType types) { lock (_lock) { - _types.UnionWith(types); + _types.UnionWith(types.Where(t => !this.Equals(t))); return this; } } @@ -158,5 +162,16 @@ public static IPythonType Combine(IPythonType x, IPythonType y) { return utx == null ? uty.Add(x) : utx.Add(uty); } + + public override bool Equals(object obj) { + if (obj is PythonUnionType u) { + lock (_lock) { + return _types.SetEquals(u._types); + } + } + return false; + } + + public override int GetHashCode() => 0; } } diff --git a/src/Analysis/Ast/Impl/Utilities/ValueEnumerator.cs b/src/Analysis/Ast/Impl/Utilities/ValueEnumerator.cs index a1d4cb7d5..7bc69a16c 100644 --- a/src/Analysis/Ast/Impl/Utilities/ValueEnumerator.cs +++ b/src/Analysis/Ast/Impl/Utilities/ValueEnumerator.cs @@ -1,13 +1,32 @@ -using System.Linq; +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Linq; using Microsoft.Python.Analysis.Specializations.Typing; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core; +using Microsoft.Python.Core.Collections; namespace Microsoft.Python.Analysis.Utilities { internal sealed class ValueEnumerator { private readonly IMember _value; private readonly IPythonType _unknown; - private readonly IMember[] _values; + private readonly IPythonModule _declaringModule; + private readonly ImmutableArray _values; + private readonly ImmutableArray _nested; private int _index; /// @@ -15,36 +34,50 @@ internal sealed class ValueEnumerator { /// /// Collection to iterate over /// Default type when we cannot find type from collection - public ValueEnumerator(IMember value, IPythonType unknown) { + public ValueEnumerator(IMember value, IPythonType unknown, IPythonModule declaringModule) { _value = value; _unknown = unknown; + _declaringModule = declaringModule; + + if (value.GetPythonType() is IPythonUnionType union) { + _nested = union.Select(v => new ValueEnumerator(v.CreateInstance(ArgumentSet.WithoutContext), unknown, declaringModule)).ToImmutableArray(); + return; + } + switch (value) { // Tuple = 'tuple value' (such as from callable). Transfer values. case IPythonCollection seq: - _values = seq.Contents.ToArray(); + _values = seq.Contents.ToImmutableArray(); break; // Create singleton list of value when cannot identify tuple default: - _values = new[] { value }; + _values = ImmutableArray.Create(value); break; } } - public IMember Next { - get { - IMember t = Peek; - _index++; - return t; + public IMember Next() { + IMember t = Peek; + _index++; + + foreach (var ve in _nested) { + ve.Next(); } + + return t; } public IMember Peek { get { - if (_values.Length > 0) { - return _index < _values.Length ? _values[_index] : _values[_values.Length - 1]; - } else { - return Filler.CreateInstance(ArgumentSet.WithoutContext); + if (_nested.Count > 0) { + return new PythonUnionType(_nested.Select(ve => ve.Peek.GetPythonType()), _declaringModule).CreateInstance(ArgumentSet.WithoutContext); } + + if (_values.Count > 0) { + return _index < _values.Count ? _values[_index] : _values[_values.Count - 1]; + } + + return Filler.CreateInstance(ArgumentSet.WithoutContext); } } diff --git a/src/Analysis/Ast/Impl/Values/Definitions/IScope.cs b/src/Analysis/Ast/Impl/Values/Definitions/IScope.cs index 7b7198783..6b350178d 100644 --- a/src/Analysis/Ast/Impl/Values/Definitions/IScope.cs +++ b/src/Analysis/Ast/Impl/Values/Definitions/IScope.cs @@ -48,6 +48,11 @@ public interface IScope { /// IReadOnlyList Children { get; } + /// + /// Locates child scope by name. + /// + IScope GetChildScope(ScopeStatement node); + /// /// Enumerates scopes from this one to global scope. /// diff --git a/src/Analysis/Ast/Impl/Values/EmptyGlobalScope.cs b/src/Analysis/Ast/Impl/Values/EmptyGlobalScope.cs index 60aa3e46c..398cf5258 100644 --- a/src/Analysis/Ast/Impl/Values/EmptyGlobalScope.cs +++ b/src/Analysis/Ast/Impl/Values/EmptyGlobalScope.cs @@ -32,6 +32,7 @@ public EmptyGlobalScope(IPythonModule module) { public IScope OuterScope => null; public IGlobalScope GlobalScope { get; } public IReadOnlyList Children => Array.Empty(); + public IScope GetChildScope(ScopeStatement node) => null; public IEnumerable EnumerateTowardsGlobal => Enumerable.Repeat(this, 1); public IEnumerable EnumerateFromGlobal => Enumerable.Repeat(this, 1); public IVariableCollection Variables => VariableCollection.Empty; diff --git a/src/Analysis/Ast/Impl/Values/PythonNone.cs b/src/Analysis/Ast/Impl/Values/PythonNone.cs new file mode 100644 index 000000000..3af3360d8 --- /dev/null +++ b/src/Analysis/Ast/Impl/Values/PythonNone.cs @@ -0,0 +1,33 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values.Collections; + +namespace Microsoft.Python.Analysis.Values { + internal sealed class PythonNone : PythonType, IPythonInstance { + public PythonNone(IBuiltinsPythonModule builtins) : base("None", new Location(builtins), string.Empty, BuiltinTypeId.None) { } + + public override IMember CreateInstance(IArgumentSet args) => this; + + public IPythonType Type => this; + + public IMember Call(string memberName, IArgumentSet args) => DeclaringModule.Interpreter.UnknownType; + + public IPythonIterator GetIterator() => new EmptyIterator(DeclaringModule.Interpreter.UnknownType); + + public IMember Index(IArgumentSet args) => DeclaringModule.Interpreter.UnknownType; + } +} diff --git a/src/Analysis/Ast/Impl/Values/Scope.cs b/src/Analysis/Ast/Impl/Values/Scope.cs index 6150e812b..7ef4408fd 100644 --- a/src/Analysis/Ast/Impl/Values/Scope.cs +++ b/src/Analysis/Ast/Impl/Values/Scope.cs @@ -30,7 +30,7 @@ internal class Scope : IScope { private VariableCollection _nonLocals; private VariableCollection _globals; private VariableCollection _imported; - private List _childScopes; + private Dictionary _childScopes; public Scope(ScopeStatement node, IScope outerScope, IPythonModule module) { OuterScope = outerScope; @@ -47,7 +47,8 @@ public Scope(ScopeStatement node, IScope outerScope, IPythonModule module) { public IScope OuterScope { get; } public IPythonModule Module { get; } - public IReadOnlyList Children => _childScopes?.ToArray() ?? Array.Empty(); + public IReadOnlyList Children => _childScopes?.Values.ToArray() ?? Array.Empty(); + public IScope GetChildScope(ScopeStatement node) => _childScopes != null && _childScopes.TryGetValue(node, out var s) ? s : null; public IVariableCollection Variables => _variables ?? VariableCollection.Empty; public IVariableCollection NonLocals => _nonLocals ?? VariableCollection.Empty; public IVariableCollection Globals => _globals ?? VariableCollection.Empty; @@ -91,7 +92,13 @@ public void DeclareImported(string name, IMember value, Location location = defa => (_imported ?? (_imported = new VariableCollection())).DeclareVariable(name, value, VariableSource.Import, location); #endregion - internal void AddChildScope(Scope s) => (_childScopes ?? (_childScopes = new List())).Add(s); + internal void AddChildScope(Scope s) + => (_childScopes ?? (_childScopes = new Dictionary()))[s.Node] = s; + + internal void ReplaceVariable(IVariable v) { + VariableCollection.RemoveVariable(v.Name); + VariableCollection.DeclareVariable(v.Name, v.Value, v.Source, v.Location); + } private VariableCollection VariableCollection => _variables ?? (_variables = new VariableCollection()); @@ -110,16 +117,18 @@ private void DeclareBuiltinVariables() { var dictType = Module.Interpreter.GetBuiltinType(BuiltinTypeId.Dict); var tupleType = Module.Interpreter.GetBuiltinType(BuiltinTypeId.Tuple); - VariableCollection.DeclareVariable("__closure__", tupleType, VariableSource.Builtin, location); - VariableCollection.DeclareVariable("__code__", objType, VariableSource.Builtin, location); - VariableCollection.DeclareVariable("__defaults__", tupleType, VariableSource.Builtin, location); - VariableCollection.DeclareVariable("__dict__", dictType, VariableSource.Builtin, location); - VariableCollection.DeclareVariable("__doc__", strType, VariableSource.Builtin, location); - VariableCollection.DeclareVariable("__func__", objType, VariableSource.Builtin, location); - VariableCollection.DeclareVariable("__globals__", dictType, VariableSource.Builtin, location); + DeclareBuiltinVariable("__closure__", tupleType, location); + DeclareBuiltinVariable("__code__", objType, location); + DeclareBuiltinVariable("__defaults__", tupleType, location); + DeclareBuiltinVariable("__dict__", dictType, location); + DeclareBuiltinVariable("__doc__", strType, location); + DeclareBuiltinVariable("__func__", objType, location); + DeclareBuiltinVariable("__globals__", dictType, location); } else if (Node is ClassDefinition) { - VariableCollection.DeclareVariable("__self__", objType, VariableSource.Builtin, location); + DeclareBuiltinVariable("__self__", objType, location); } } + protected void DeclareBuiltinVariable(string name, IPythonType type, Location location) + => VariableCollection.DeclareVariable(name, type, VariableSource.Builtin, location); } } diff --git a/src/Analysis/Ast/Impl/get_search_paths.py b/src/Analysis/Ast/Impl/get_search_paths.py index 49cd42e2e..36a56c43d 100644 --- a/src/Analysis/Ast/Impl/get_search_paths.py +++ b/src/Analysis/Ast/Impl/get_search_paths.py @@ -44,6 +44,7 @@ def clean(path): BEFORE_SITE = set(clean(p) for p in BEFORE_SITE) AFTER_SITE = set(clean(p) for p in AFTER_SITE) +SCRIPT_DIR = clean(os.path.dirname(os.path.realpath(__file__))) try: SITE_PKGS = set(clean(p) for p in site.getsitepackages()) @@ -72,9 +73,12 @@ def clean(path): for p in sys.path: p = clean(p) + + if p == SCRIPT_DIR or p.startswith(SCRIPT_DIR + os.sep): + continue if not os.path.isdir(p) and not (os.path.isfile(p) and zipfile.is_zipfile(p)): - continue + continue if p in BEFORE_SITE: print("%s|stdlib|" % p) diff --git a/src/Analysis/Ast/Impl/scrape_module.py b/src/Analysis/Ast/Impl/scrape_module.py index 9432a6aed..6f3270ac2 100644 --- a/src/Analysis/Ast/Impl/scrape_module.py +++ b/src/Analysis/Ast/Impl/scrape_module.py @@ -309,6 +309,9 @@ def __init__(self, # We have a property self.decorators = '@property', self.fullsig = self.name + "(" + ", ".join(self._defaults) + ")" + + if scope_alias == "__Object__" and name == "__init__": + self.fullsig = "__init__(self)" self.fullsig = ( self.fullsig or @@ -1408,8 +1411,9 @@ def add_type(alias, type_obj): state.members.append(mi) state.members.append(MemberInfo(alias, None, literal=mi.name)) - add_simple('__Unknown__', '', MemberInfo("__name__", None, literal='""')) - add_simple('__NoneType__', 'the type of the None object', MemberInfo.NO_VALUE) + # We don't need these anymore, as Unknown and None are specialized in the analysis. + #add_simple('__Unknown__', '', MemberInfo("__name__", None, literal='""')) + #add_simple('__NoneType__', 'the type of the None object', MemberInfo.NO_VALUE) # NoneType and None are explicitly defined to avoid parser errors # because of None being a keyword. diff --git a/src/Analysis/Ast/Test/AnalysisTestBase.cs b/src/Analysis/Ast/Test/AnalysisTestBase.cs index 89d1fa046..db915ae7f 100644 --- a/src/Analysis/Ast/Test/AnalysisTestBase.cs +++ b/src/Analysis/Ast/Test/AnalysisTestBase.cs @@ -20,6 +20,7 @@ using System.Threading.Tasks; using FluentAssertions; using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Analysis.Caching; using Microsoft.Python.Analysis.Core.Interpreter; using Microsoft.Python.Analysis.Diagnostics; using Microsoft.Python.Analysis.Documents; @@ -37,6 +38,12 @@ namespace Microsoft.Python.Analysis.Tests { public abstract class AnalysisTestBase { + protected const int AnalysisTimeoutInMS = 1000 * 60; + + protected TimeSpan AnalysisTimeout { get; set; } = TimeSpan.FromMilliseconds(AnalysisTimeoutInMS); + + private TimeSpan GetAnalysisTimeout() => Debugger.IsAttached ? Timeout.InfiniteTimeSpan : AnalysisTimeout; + protected TestLogger TestLogger { get; } = new TestLogger(); protected ServiceManager Services { get; private set; } @@ -79,7 +86,9 @@ protected async Task CreateServicesAsync(string root, Interpret } TestLogger.Log(TraceEventType.Information, "Create PythonAnalyzer"); - var analyzer = new PythonAnalyzer(sm, stubCacheFolderPath); + + CacheService.Register(sm, stubCacheFolderPath, pathCheck: false); + var analyzer = new PythonAnalyzer(sm); sm.AddService(analyzer); TestLogger.Log(TraceEventType.Information, "Create PythonInterpreter"); @@ -152,8 +161,13 @@ protected async Task GetAnalysisAsync( TestLogger.Log(TraceEventType.Information, "Test: AST end."); TestLogger.Log(TraceEventType.Information, "Test: Analysis begin."); - await services.GetService().WaitForCompleteAnalysisAsync(); - var analysis = await doc.GetAnalysisAsync(-1); + + IDocumentAnalysis analysis; + using (var cts = new CancellationTokenSource(GetAnalysisTimeout())) { + await services.GetService().WaitForCompleteAnalysisAsync(cts.Token); + analysis = await doc.GetAnalysisAsync(-1, cts.Token); + } + analysis.Should().NotBeNull(); TestLogger.Log(TraceEventType.Information, "Test: Analysis end."); @@ -162,8 +176,10 @@ protected async Task GetAnalysisAsync( protected async Task GetDocumentAnalysisAsync(IDocument document) { var analyzer = Services.GetService(); - await analyzer.WaitForCompleteAnalysisAsync(); - return await document.GetAnalysisAsync(Timeout.Infinite); + using (var cts = new CancellationTokenSource(GetAnalysisTimeout())) { + await analyzer.WaitForCompleteAnalysisAsync(cts.Token); + return await document.GetAnalysisAsync(Timeout.Infinite, cts.Token); + } } } } diff --git a/src/Analysis/Ast/Test/ArgumentSetTests.cs b/src/Analysis/Ast/Test/ArgumentSetTests.cs index 24cf057f0..e4dc8e2b5 100644 --- a/src/Analysis/Ast/Test/ArgumentSetTests.cs +++ b/src/Analysis/Ast/Test/ArgumentSetTests.cs @@ -166,59 +166,6 @@ def f(a, b, **dict): ... argSet.DictionaryArgument.Expressions["e"].Should().BeAssignableTo().Which.Value.Should().Be("str"); } - [TestMethod, Priority(0)] - public async Task TypeVarSpecializedArgs() { - const string code = @" -from typing import TypeVar - -TypeVar('T', int, float, str, bound='test', covariant=True) -"; - var argSet = await GetArgSetAsync(code, funcName: "TypeVar"); - argSet.Arguments.Count.Should().Be(4); - - argSet.Arguments[0].Name.Should().Be("name"); - argSet.Arguments[0].ValueExpression.Should().BeOfType().Which.Value.Should().Be("T"); - argSet.Arguments[1].Name.Should().Be("bound"); - argSet.Arguments[1].ValueExpression.Should().BeOfType().Which.Value.Should().Be("test"); - argSet.Arguments[2].Name.Should().Be("covariant"); - argSet.Arguments[2].ValueExpression.Should().BeOfType().Which.Value.Should().Be(true); - argSet.Arguments[3].Name.Should().Be("contravariant"); - argSet.Arguments[3].Value.Should().BeOfType().Which.Value.Should().Be(false); - - argSet.ListArgument.Should().NotBeNull(); - argSet.ListArgument.Name.Should().Be("constraints"); - argSet.ListArgument.Expressions.OfType().Select(c => c.Name).Should().ContainInOrder("int", "float", "str"); - - argSet.DictionaryArgument.Should().BeNull(); - } - - [TestMethod, Priority(0)] - public async Task TypeVarDefaultValues() { - const string code = @" -from typing import TypeVar - -TypeVar('T', int, float, str) -"; - var argSet = await GetArgSetAsync(code, funcName: "TypeVar"); - argSet.Arguments.Count.Should().Be(4); - - argSet.Arguments[0].Name.Should().Be("name"); - argSet.Arguments[0].ValueExpression.Should().BeOfType().Which.Value.Should().Be("T"); - argSet.Arguments[1].Name.Should().Be("bound"); - argSet.Arguments[1].Value.Should().BeOfType().Which.Value.Should().Be(null); - argSet.Arguments[2].Name.Should().Be("covariant"); - argSet.Arguments[2].Value.Should().BeOfType().Which.Value.Should().Be(false); - argSet.Arguments[3].Name.Should().Be("contravariant"); - argSet.Arguments[3].Value.Should().BeOfType().Which.Value.Should().Be(false); - - argSet.ListArgument.Should().NotBeNull(); - argSet.ListArgument.Name.Should().Be("constraints"); - argSet.ListArgument.Expressions.OfType().Select(c => c.Name).Should().ContainInOrder("int", "float", "str"); - - argSet.DictionaryArgument.Should().BeNull(); - } - - [TestMethod, Priority(0)] public async Task NewTypeSpecializedArgs() { const string code = @" @@ -437,6 +384,42 @@ def func(a = A()): ... t.MemberType.Should().Be(PythonMemberType.Class); } + [TestMethod, Priority(0)] + public async Task PositionalOnly() { + const string code = @" +def f(a, b, /, c, d, *, e, f): ... +f(1, 2, 3, 4, e='a', f=False) +"; + var argSet = await GetArgSetAsync(code); + argSet.Arguments.Count.Should().Be(6); + argSet.Arguments[0].Name.Should().Be("a"); + argSet.Arguments[0].ValueExpression.Should().BeOfType().Which.Value.Should().Be(1); + argSet.Arguments[1].Name.Should().Be("b"); + argSet.Arguments[1].ValueExpression.Should().BeOfType().Which.Value.Should().Be(2); + argSet.Arguments[2].Name.Should().Be("c"); + argSet.Arguments[2].ValueExpression.Should().BeOfType().Which.Value.Should().Be(3); + argSet.Arguments[3].Name.Should().Be("d"); + argSet.Arguments[3].ValueExpression.Should().BeOfType().Which.Value.Should().Be(4); + argSet.Arguments[4].Name.Should().Be("e"); + argSet.Arguments[4].ValueExpression.Should().BeOfType().Which.Value.Should().Be("a"); + argSet.Arguments[5].Name.Should().Be("f"); + argSet.Arguments[5].ValueExpression.Should().BeOfType().Which.Value.Should().Be(false); + argSet.ListArgument.Should().BeNull(); + argSet.DictionaryArgument.Should().BeNull(); + } + + [TestMethod, Priority(0)] + public async Task PositionalOnlyNamed() { + const string code = @" +def f(a, /): ... +f(a=1) +"; + var argSet = await GetArgSetAsync(code); + argSet.Arguments.Count.Should().Be(1); + argSet.Errors.Count.Should().Be(1); + argSet.Errors[0].ErrorCode.Should().Be(ErrorCodes.PositionalOnlyNamed); + } + private async Task GetArgSetAsync(string code, string funcName = "f") { var analysis = await GetAnalysisAsync(code); var f = analysis.Should().HaveFunction(funcName).Which; diff --git a/src/Analysis/Ast/Test/AssignmentTests.cs b/src/Analysis/Ast/Test/AssignmentTests.cs index bb7e000e8..1f7321d94 100644 --- a/src/Analysis/Ast/Test/AssignmentTests.cs +++ b/src/Analysis/Ast/Test/AssignmentTests.cs @@ -219,6 +219,15 @@ def __init__(self): .Which.Should().HaveMembers("abc", "y", "__class__"); } + [TestMethod, Priority(0)] + public async Task AugmentedAssignToUndefined() { + const string code = @" +x += 1 +"; + var analysis = await GetAnalysisAsync(code); + analysis.Should().NotHaveVariable("x"); + } + [TestMethod, Priority(0)] public async Task BaseInstanceVariable() { const string code = @" @@ -674,8 +683,7 @@ public async Task NamedExpressionIf() { if (x := 1234) == 1234: pass "; - var analysis = await GetAnalysisAsync(code); - + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.Int); } @@ -687,8 +695,17 @@ from typing import List a: List[int] b = [(x := i) for i in a] "; - var analysis = await GetAnalysisAsync(code); + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); + analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.Int); + } + [TestMethod, Priority(0)] + public async Task WalrusInWhile() { + const string code = @" +while(x := 1): + y = x +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.Int); } } diff --git a/src/Analysis/Ast/Test/ClassesTests.cs b/src/Analysis/Ast/Test/ClassesTests.cs index 61e3f6f94..d19407e24 100644 --- a/src/Analysis/Ast/Test/ClassesTests.cs +++ b/src/Analysis/Ast/Test/ClassesTests.cs @@ -270,6 +270,23 @@ def __init__(self, value): .Which.Should().HaveParameterAt(0).Which.Should().HaveName("self").And.HaveType("X"); } + [TestMethod, Priority(0)] + public async Task ClassBaseInit() { + const string code = @" +class A: + def __init__(self, value): + self.value = value + +class B(A): ... +"; + var analysis = await GetAnalysisAsync(code); + analysis.Should().HaveClass("B") + .Which.Should().HaveMethod("__init__") + .Which.Should().HaveSingleOverload() + .Which.Should().HaveParameterAt(1) + .Which.Name.Should().Be("value"); + } + [TestMethod, Priority(0)] public async Task ClassBase2X() { const string code = @" @@ -834,5 +851,28 @@ from Base import A var analysis = await GetAnalysisAsync(code); analysis.Should().HaveVariable("x").OfType("A").Which.Should().HaveMember("methodABase"); } + + [TestMethod, Priority(0)] + public async Task InnerClassAsClassMember() { + const string code = @" +class test(): + class Test2(): + def Z(self): + return + A = Test2() + + def X(self) -> Test2: + return Test2() +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); + var test = analysis.Should().HaveClass("test").Which; + + test.Should().HaveMember("A").Which + .Should().HaveMethod("Z"); + + test.Should().HaveMethod("X").Which + .Should().HaveSingleOverload().Which + .Should().HaveReturnType("Test2"); + } } } diff --git a/src/Analysis/Ast/Test/DocumentBufferTests.cs b/src/Analysis/Ast/Test/DocumentBufferTests.cs index c0230ac18..2743237cc 100644 --- a/src/Analysis/Ast/Test/DocumentBufferTests.cs +++ b/src/Analysis/Ast/Test/DocumentBufferTests.cs @@ -29,7 +29,7 @@ public class DocumentBufferTests { [TestMethod, Priority(0)] public void BasicDocumentBuffer() { var doc = new DocumentBuffer(); - doc.Reset(0, @"def f(x): + doc.SetContent(@"def f(x): return def g(y): @@ -77,7 +77,7 @@ def g(y): public void ResetDocumentBuffer() { var doc = new DocumentBuffer(); - doc.Reset(0, string.Empty); + doc.SetContent(string.Empty); Assert.AreEqual(string.Empty, doc.Text); doc.Update(new[] { @@ -86,18 +86,13 @@ public void ResetDocumentBuffer() { Assert.AreEqual("text", doc.Text); Assert.AreEqual(1, doc.Version); - - doc.Reset(0, @"abcdef"); - - Assert.AreEqual(@"abcdef", doc.Text); - Assert.AreEqual(0, doc.Version); } [TestMethod, Priority(0)] public void ReplaceAllDocumentBuffer() { var doc = new DocumentBuffer(); - doc.Reset(0, string.Empty); + doc.SetContent(string.Empty); Assert.AreEqual(string.Empty, doc.Text); doc.Update(new[] { @@ -134,7 +129,7 @@ public void ReplaceAllDocumentBuffer() { [TestMethod, Priority(0)] public void DeleteMultipleDisjoint() { var doc = new DocumentBuffer(); - doc.Reset(0, @" + doc.SetContent(@" line1 line2 line3 @@ -157,7 +152,7 @@ public void DeleteMultipleDisjoint() { [TestMethod, Priority(0)] public void InsertMultipleDisjoint() { var doc = new DocumentBuffer(); - doc.Reset(0, @" + doc.SetContent(@" line line line @@ -180,7 +175,7 @@ public void InsertMultipleDisjoint() { [TestMethod, Priority(0)] public void DeleteAcrossLines() { var doc = new DocumentBuffer(); - doc.Reset(0, @" + doc.SetContent(@" line1 line2 line3 @@ -199,7 +194,7 @@ public void DeleteAcrossLines() { [TestMethod, Priority(0)] public void SequentialChanges() { var doc = new DocumentBuffer(); - doc.Reset(0, @" + doc.SetContent(@" line1 line2 line3 @@ -218,7 +213,7 @@ public void SequentialChanges() { [TestMethod, Priority(0)] public void InsertTopToBottom() { var doc = new DocumentBuffer(); - doc.Reset(0, @"linelinelineline"); + doc.SetContent(@"linelinelineline"); doc.Update(new[] { DocumentChange.Insert("\n", new SourceLocation(1, 1)), DocumentChange.Insert("1\n", new SourceLocation(2, 5)), @@ -233,7 +228,7 @@ public void InsertTopToBottom() { [DataTestMethod, Priority(0)] public void NewLines(string s, NewLineLocation[] expected) { var doc = new DocumentBuffer(); - doc.Reset(0, s); + doc.SetContent(s); var nls = doc.GetNewLineLocations().ToArray(); for (var i = 0; i < nls.Length; i++) { Assert.AreEqual(nls[i].Kind, expected[i].Kind); diff --git a/src/Analysis/Ast/Test/FluentAssertions/MemberAssertions.cs b/src/Analysis/Ast/Test/FluentAssertions/MemberAssertions.cs index 871e0a37a..1f6b10c08 100644 --- a/src/Analysis/Ast/Test/FluentAssertions/MemberAssertions.cs +++ b/src/Analysis/Ast/Test/FluentAssertions/MemberAssertions.cs @@ -83,6 +83,20 @@ public AndWhichConstraint HaveBase(string name, s public AndWhichConstraint HaveMethod(string name, string because = "", params object[] reasonArgs) => HaveMember(name, because, reasonArgs).OfMemberType(PythonMemberType.Method); + public void HaveMemberName(string name, string because = "", params object[] reasonArgs) { + NotBeNull(); + + var t = Subject.GetPythonType(); + var mc = (IMemberContainer)t; + Execute.Assertion.ForCondition(mc != null) + .BecauseOf(because, reasonArgs) + .FailWith($"Expected {GetName(t)} to be a member container{{reason}}."); + + Execute.Assertion.ForCondition(mc.GetMemberNames().Contains(name)) + .BecauseOf(because, reasonArgs) + .FailWith($"Expected {GetName(t)} to have a member named '{name}'{{reason}}."); + } + public AndWhichConstraint HaveMember(string name, string because = "", params object[] reasonArgs) where TMember : class, IMember { @@ -125,7 +139,7 @@ public void HaveSameMembersAs(IMember other) { Debug.Assert(missingNames.Length == 0); missingNames.Should().BeEmpty("Subject has missing names: ", missingNames); - + Debug.Assert(extraNames.Length == 0); extraNames.Should().BeEmpty("Subject has extra names: ", extraNames); @@ -147,7 +161,7 @@ public void HaveSameMembersAs(IMember other) { var otherClass = otherMemberType as IPythonClassType; otherClass.Should().NotBeNull(); - if(subjectClass is IGenericType gt) { + if (subjectClass is IGenericType gt) { otherClass.Should().BeAssignableTo(); otherClass.IsGeneric.Should().Be(gt.IsGeneric, $"Class name: {subjectClass.Name}"); } diff --git a/src/Analysis/Ast/Test/FluentAssertions/VariableAssertions.cs b/src/Analysis/Ast/Test/FluentAssertions/VariableAssertions.cs index 6f0aac0c1..fff01147f 100644 --- a/src/Analysis/Ast/Test/FluentAssertions/VariableAssertions.cs +++ b/src/Analysis/Ast/Test/FluentAssertions/VariableAssertions.cs @@ -65,8 +65,7 @@ public AndWhichConstraint HaveMember(string name, s public AndWhichConstraint NotHaveMember(string name, string because = "", params object[] reasonArgs) { NotBeNull(because, reasonArgs); - var m = Value.GetPythonType().GetMember(name); - m.GetPythonType().IsUnknown().Should().BeTrue(); + Value.GetPythonType().GetMemberNames().Should().NotContain(name); return new AndWhichConstraint(this, Subject); } @@ -90,6 +89,11 @@ public AndWhichConstraint HaveMember(string name, stri return new AndWhichConstraint(this, m); } + public void HaveMemberName(string name, string because = "", params object[] reasonArgs) { + NotBeNull(because, reasonArgs); + Value.GetPythonType().GetMemberNames().Should().Contain(name); + } + public AndWhichConstraint HaveOverloadWithParametersAt(int index, string because = "", params object[] reasonArgs) { var constraint = HaveOverloadAt(index); var overload = constraint.Which; diff --git a/src/Analysis/Ast/Test/FunctionTests.cs b/src/Analysis/Ast/Test/FunctionTests.cs index f8c4e8928..e75dbcb2b 100644 --- a/src/Analysis/Ast/Test/FunctionTests.cs +++ b/src/Analysis/Ast/Test/FunctionTests.cs @@ -115,11 +115,11 @@ def f(s: s = 123): return s "; var analysis = await GetAnalysisAsync(code); - analysis.Should().HaveVariable("s").OfType(BuiltinTypeId.NoneType); + analysis.Should().HaveVariable("s").OfType(BuiltinTypeId.None); analysis.Should().HaveFunction("f") .Which.Should().HaveSingleOverload() .Which.Should().HaveSingleParameter() - .Which.Should().HaveName("s").And.HaveType(BuiltinTypeId.NoneType); + .Which.Should().HaveName("s").And.HaveType(BuiltinTypeId.None); } [TestMethod, Priority(0)] @@ -130,7 +130,7 @@ def f(s: lambda s: s > 0 = 123): return s "; var analysis = await GetAnalysisAsync(code); - analysis.Should().HaveVariable("s").OfType(BuiltinTypeId.NoneType) + analysis.Should().HaveVariable("s").OfType(BuiltinTypeId.None) .And.HaveFunction("f") .Which.Should().HaveSingleOverload() .Which.Should().HaveSingleParameter() @@ -369,7 +369,7 @@ def f(a: str) -> bytes: ... var f = analysis.Should().HaveFunction("f").Which; f.Should().HaveOverloadAt(0) - .Which.Should().HaveReturnType(BuiltinTypeId.NoneType) + .Which.Should().HaveReturnType(BuiltinTypeId.None) .Which.Should().HaveSingleParameter() .Which.Should().HaveName("a").And.HaveType(BuiltinTypeId.Bool); @@ -383,7 +383,7 @@ def f(a: str) -> bytes: ... .Which.Should().HaveSingleParameter() .Which.Should().HaveName("a").And.HaveType(BuiltinTypeId.Str); - analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.NoneType) + analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.None) .And.HaveVariable("y").OfType(BuiltinTypeId.Float) .And.HaveVariable("z").OfType(BuiltinTypeId.Bytes); } @@ -609,12 +609,11 @@ def func(): ... class B: @deprecation.deprecated('') def b(self): pass - "; var analysis = await GetAnalysisAsync(code); - analysis.Should().NotHaveVariable("A"); - analysis.Should().NotHaveVariable("func"); - analysis.Should().HaveVariable("B").Which.Should().NotHaveMember("b"); + analysis.Should().HaveVariable("A"); + analysis.Should().HaveVariable("func"); + analysis.Should().HaveVariable("B").Which.Should().HaveMember("b"); } [TestMethod, Priority(0)] @@ -657,5 +656,31 @@ def A(self) -> int: .Which.Should().HaveType("A") .Which.Should().BeOfType(a.GetType()); } + + [TestMethod, Priority(0)] + public async Task PositionalOnlyParameters() { + const string code = @" +def f(a, b, /, c, d): + pass +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); + analysis.Should().HaveFunction("f") + .Which.Should().HaveSingleOverload() + .Which.Should().HaveParameters("a", "b", "c", "d"); + } + + [TestMethod, Priority(0)] + public async Task LiteralParameter() { + const string code = @" +from typing import Literal + +def f(handle: int, overlapped: Literal[True]): + pass +"; + var analysis = await GetAnalysisAsync(code); + analysis.Should().HaveFunction("f") + .Which.Should().HaveSingleOverload() + .Which.Should().HaveParameters("handle", "overlapped"); + } } } diff --git a/src/Analysis/Ast/Test/ImportTests.cs b/src/Analysis/Ast/Test/ImportTests.cs index 3e28e8a1b..42f20587b 100644 --- a/src/Analysis/Ast/Test/ImportTests.cs +++ b/src/Analysis/Ast/Test/ImportTests.cs @@ -13,13 +13,18 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System.IO; using System.Linq; +using System.Threading; using System.Threading.Tasks; using FluentAssertions; +using Microsoft.Python.Analysis.Analyzer; using Microsoft.Python.Analysis.Diagnostics; +using Microsoft.Python.Analysis.Documents; using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Tests.FluentAssertions; using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; using Microsoft.Python.Core; using Microsoft.Python.Parsing.Tests; using Microsoft.VisualStudio.TestTools.UnitTesting; @@ -105,8 +110,8 @@ public async Task BuiltinImport() { var analysis = await GetAnalysisAsync(@"import sys"); analysis.Should().HaveVariable("sys") - .Which.Should().HaveType(BuiltinTypeId.Module) - .And.HaveMember("platform"); + .Which.Should().HaveType(BuiltinTypeId.Module) + .And.HaveMember("platform"); } [TestMethod, Priority(0)] @@ -214,6 +219,22 @@ public async Task UnresolvedRelativeFromImportAs() { d.Message.Should().Be(Resources.ErrorRelativeImportBeyondTopLevel.FormatInvariant("nonexistent")); } + [TestMethod, Priority(0)] + public async Task UnresolvedImportInTry() { + var analysis = await GetAnalysisAsync(@" +def foo(): + try: + import nonexistent + except: + pass +"); + analysis.Diagnostics.Should().HaveCount(1); + var d = analysis.Diagnostics.First(); + d.ErrorCode.Should().Be(ErrorCodes.UnresolvedImport); + d.SourceSpan.Should().Be(4, 16, 4, 27); + d.Message.Should().Be(Resources.ErrorUnresolvedImport.FormatInvariant("nonexistent")); + } + [TestMethod, Priority(0)] public async Task FromFuture() { var analysis = await GetAnalysisAsync(@"from __future__ import print_function", PythonVersions.LatestAvailable2X); @@ -234,5 +255,144 @@ def exit(): var analysis = await GetAnalysisAsync(code); analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.Int); } + + [TestMethod, Priority(0)] + public async Task ModuleInternalImportSys() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "__init__.py"), "from . import m1\nimport sys"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "m1", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "import package"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + analysis.Should().HaveVariable("package").Which.Should().HaveMembers("m1", "sys"); + } + + [TestMethod, Priority(0)] + public async Task ModuleImportingSubmodule() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "__init__.py"), "import package.m1"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "m1", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "import package"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + analysis.Should().HaveVariable("package").Which.Should().HaveMember("m1"); + } + + [TestMethod, Priority(0)] + public async Task ModuleImportingSubmodules() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "__init__.py"), @" +from top import sub1 +import top.sub2 +import top.sub3.sub4 +"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub2.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub3", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub3", "sub4.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "import top"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + analysis.Should().HaveVariable("top").Which.Should().HaveMembers("sub1", "sub2", "sub3", "top"); + } + + + [TestMethod, Priority(0)] + public async Task ImportPackageNoInitPy() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "from top import sub1"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + var sub1 = analysis.Should().HaveVariable("sub1") + .Which.Should().HaveType().Which; + sub1.Value.MemberType.Should().NotBe(ModuleType.Unresolved); + } + + [TestMethod, Priority(0)] + public async Task DeepSubmoduleImport() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "sub2", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "sub2", "sub3", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "sub2", "sub3", "sub4", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var appDoc = rdt.OpenDocument(appUri, "import top.sub1.sub2.sub3.sub4"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await appDoc.GetAnalysisAsync(Timeout.Infinite); + + var topModule = analysis.Should().HaveVariable("top") + .Which.Should().HaveType().Which; + + topModule.Should().HaveMemberName("sub1"); + var sub1Module = topModule.Should().HaveMember("sub1").Which; + + sub1Module.Should().HaveMemberName("sub2"); + var sub2Module = sub1Module.Should().HaveMember("sub2").Which; + + sub2Module.Should().HaveMemberName("sub3"); + var sub3Module = sub2Module.Should().HaveMember("sub3").Which; + + sub3Module.Should().HaveMemberName("sub4"); + sub3Module.Should().HaveMember("sub4"); + } + + [TestMethod, Priority(0)] + public async Task SubmoduleOverridesVariable() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "__init__.py"), "sub2 = 1"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "sub2", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var appDoc = rdt.OpenDocument(appUri, "from top.sub1 import sub2"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await appDoc.GetAnalysisAsync(Timeout.Infinite); + analysis.Should().HaveVariable("sub2").Which.Should().HaveType(BuiltinTypeId.Module); + } + + [TestMethod, Priority(0)] + public async Task SubmoduleOverridesVariableStarImport() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "__init__.py"), "sub2 = 1"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "sub2", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var appDoc = rdt.OpenDocument(appUri, "from top.sub1 import *"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await appDoc.GetAnalysisAsync(Timeout.Infinite); + analysis.Should().HaveVariable("sub2").Which.Should().HaveType(BuiltinTypeId.Module); + } + + [TestMethod, Priority(0)] + public async Task CircularDependencyFunctools() { + var analysis = await GetAnalysisAsync(@"from _functools import partial"); + analysis.Should().HaveClass("partial"); + } } } diff --git a/src/Analysis/Ast/Test/InheritanceTests.cs b/src/Analysis/Ast/Test/InheritanceTests.cs index 669b4f680..9cd5b8206 100644 --- a/src/Analysis/Ast/Test/InheritanceTests.cs +++ b/src/Analysis/Ast/Test/InheritanceTests.cs @@ -113,5 +113,247 @@ def __new__(cls): .Which.Value.Should().BeAssignableTo() .Which.Type.Name.Should().Be("A"); } + + [TestMethod, Priority(0)] + public async Task SuperShouldReturnBaseClassFunctions() { + const string code = @" +class Baze: + def base_func(self): + return 1234 + +class Derived(Baze): + def foo(self): + x = super() +"; + + var analysis = await GetAnalysisAsync(code); + + // the class, for which we know parameter type initially + analysis.Should().HaveClass("Derived").Which.Should().HaveMethod("foo") + .Which.Should().HaveVariable("x") + .Which.Value.Should().HaveMemberName("base_func"); + } + + + [TestMethod, Priority(0)] + public async Task SuperShouldReturnBaseClassFunctionsWhenCalledWithSelf() { + const string code = @" +class Baze: + def base_func(self): + return 1234 + +class Derived(Baze): + def foo(self): + x = super(Derived, self) +"; + + var analysis = await GetAnalysisAsync(code); + + // the class, for which we know parameter type initially + analysis.Should().HaveClass("Derived").Which.Should().HaveMethod("foo") + .Which.Should().HaveVariable("x") + .Which.Value.Should().HaveMemberName("base_func"); + } + + + [TestMethod, Priority(0)] + public async Task SuperWithSecondParamDerivedClassShouldOnlyHaveBaseMembers() { + const string code = @" +class Baze: + def baze_foo(self): + pass + +class Derived(Baze): + def foo(self): + pass + +d = Derived() + +x = super(Derived, d) +"; + + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveVariable("x") + .Which.Value.Should().HaveMemberName("baze_foo"); + + analysis.Should().HaveVariable("x") + .Which.Value.Should().NotHaveMembers("foo"); + } + + [TestMethod, Priority(0)] + public async Task SuperWithSecondParamParentShouldOnlyReturnGrandparentMembers() { + const string code = @" +class A: + def a_foo(self): + pass + +class B(A): + def b_foo(self): + pass + +class C(B): + def c_foo(self): + pass + +c = C() + +x = super(B, c) # super starts its search after 'B' in the mro +"; + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveVariable("x").Which.Value + .Should().HaveMembers("a_foo") + .And.NotHaveMembers("b_foo") + .And.NotHaveMembers("c_foo"); + } + + + [TestMethod, Priority(0)] + public async Task SuperWithSecondParamInvalidShouldBeUnknown() { + const string code = @" +class A: + def a_foo(self): + pass + +class B(A): + def b_foo(self): + pass + +x = super(B, DummyClass) # super starts its search after 'b' +"; + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.Unknown); + } + + [TestMethod, Priority(0)] + public async Task SuperWithFirstParamInvalid() { + const string code = @" +class A: + def a_foo(self): + pass + +class B(A): + def b_foo(self): + pass + +x = super(DummyClass, B) # super starts its search after 'b' +"; + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveVariable("x").OfType(BuiltinTypeId.Unknown); + } + + [TestMethod, Priority(0)] + public async Task SuperUnbounded() { + const string code = @" +class A: + def a_foo(self): + pass + +class B(A): + def b_foo(self): + pass + +x = super(B) # super starts its search after 'b' +"; + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveVariable("x").Which.Value + .Should().HaveMembers("a_foo") + .And.NotHaveMembers("b_foo"); + } + + [TestMethod, Priority(0)] + public async Task SuperWithNoBaseShouldReturnObject() { + const string code = @" +class A(): + def foo(self): + x = super() +"; + var analysis = await GetAnalysisAsync(code); + + // the class, for which we know parameter type initially + analysis.Should().HaveClass("A").Which.Should().HaveMethod("foo") + .Which.Should().HaveVariable("x").OfType(BuiltinTypeId.Type); + } + + [TestMethod, Priority(0)] + public async Task FunctionWithNoClassCallingSuperShouldFail() { + const string code = @" +def foo(self): + x = super() +"; + + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveFunction("foo") + .Which.Should().HaveVariable("x") + .Which.Name.Should().Be("x"); + } + + [TestMethod, Priority(0)] + public async Task FunctionAssigningIntToSuperShouldBeInt() { + const string code = @" +def foo(self): + super = 1 +"; + + var analysis = await GetAnalysisAsync(code); + + analysis.Should().HaveFunction("foo") + .Which.Should().HaveVariable("super") + .Which.Value.IsOfType(BuiltinTypeId.Int); + } + + [TestMethod, Priority(0)] + public async Task SuperShouldReturnAllBaseClassMembers() { + const string code = @" +class GrandParent: + def grand_func(self): + return 1 + +class Parent(GrandParent): + def parent_func(self): + return 2 + +class Child(Parent): + def child_func(self): + x = super() +"; + + var analysis = await GetAnalysisAsync(code); + + var x = analysis.Should().HaveClass("Child").Which.Should().HaveMethod("child_func") + .Which.Should().HaveVariable("x").Which; + + x.Value.Should().HaveMemberName("grand_func"); + x.Value.Should().HaveMemberName("parent_func"); + } + + + [TestMethod, Priority(0)] + public async Task SuperShouldReturnAllBaseClassMembersGenerics() { + const string code = @" +from typing import Generic, TypeVar +T = TypeVar('T') +class A(Generic[T]): + def af(self, x: T) -> T: + return x +class B(A[int]): # leave signature as is + def bf(self, x: T) -> T: + y = super() + return x +"; + + var analysis = await GetAnalysisAsync(code); + + var y = analysis.Should().HaveClass("B") + .Which.Should().HaveMethod("bf") + .Which.Should().HaveVariable("y").Which; + + y.Value.Should().HaveMemberName("af"); + } } } diff --git a/src/Analysis/Ast/Test/LintInheritNonClassTests.cs b/src/Analysis/Ast/Test/LintInheritNonClassTests.cs index 812e8d6be..91fc86c5f 100644 --- a/src/Analysis/Ast/Test/LintInheritNonClassTests.cs +++ b/src/Analysis/Ast/Test/LintInheritNonClassTests.cs @@ -489,5 +489,39 @@ class C(Enum): ... var analysis = await GetAnalysisAsync(code); analysis.Diagnostics.Should().BeEmpty(); } + + [TestMethod, Priority(0)] + public async Task InheritFromNone() { + const string code = @" +class C(None): + def method(self): + return 'test' +"; + var analysis = await GetAnalysisAsync(code); + analysis.Diagnostics.Should().HaveCount(1); + + var diagnostic = analysis.Diagnostics.ElementAt(0); + diagnostic.SourceSpan.Should().Be(2, 9, 2, 13); + diagnostic.Message.Should().Be(Resources.InheritNonClass.FormatInvariant("None")); + diagnostic.ErrorCode.Should().Be(ErrorCodes.InheritNonClass); + } + + [TestMethod, Priority(0)] + public async Task InheritFromNoneVar() { + const string code = @" +x = None + +class C(x): + def method(self): + return 'test' +"; + var analysis = await GetAnalysisAsync(code); + analysis.Diagnostics.Should().HaveCount(1); + + var diagnostic = analysis.Diagnostics.ElementAt(0); + diagnostic.SourceSpan.Should().Be(4, 9, 4, 10); + diagnostic.Message.Should().Be(Resources.InheritNonClass.FormatInvariant("x")); + diagnostic.ErrorCode.Should().Be(ErrorCodes.InheritNonClass); + } } } diff --git a/src/Analysis/Ast/Test/LintNoMethodArgumentTests.cs b/src/Analysis/Ast/Test/LintNoMethodArgumentTests.cs index d83530c29..ee16e1339 100644 --- a/src/Analysis/Ast/Test/LintNoMethodArgumentTests.cs +++ b/src/Analysis/Ast/Test/LintNoMethodArgumentTests.cs @@ -118,6 +118,16 @@ public async Task NoDiagnosticInMetaclass() { class Test(type): def test(): pass +"; + var analysis = await GetAnalysisAsync(code); + analysis.Diagnostics.Should().BeEmpty(); + } + + [TestMethod, Priority(0)] + public async Task NoDiagnosticNoParameters() { + const string code = @" +class test: + def walk "; var analysis = await GetAnalysisAsync(code); analysis.Diagnostics.Should().BeEmpty(); diff --git a/src/Analysis/Ast/Test/LintNoSelfArgumentTests.cs b/src/Analysis/Ast/Test/LintNoSelfArgumentTests.cs index abedbd8b3..af7297e43 100644 --- a/src/Analysis/Ast/Test/LintNoSelfArgumentTests.cs +++ b/src/Analysis/Ast/Test/LintNoSelfArgumentTests.cs @@ -165,6 +165,16 @@ public async Task NoDiagnosticInMetaclass() { class Test(type): def test(a): pass +"; + var analysis = await GetAnalysisAsync(code); + analysis.Diagnostics.Should().BeEmpty(); + } + + [TestMethod, Priority(0)] + public async Task LambdaInClassScope() { + const string code = @" +class C: + [lambda x, y: x+y+z for z in range(10)] "; var analysis = await GetAnalysisAsync(code); analysis.Diagnostics.Should().BeEmpty(); diff --git a/src/Analysis/Ast/Test/LintUndefinedVarsTests.cs b/src/Analysis/Ast/Test/LintUndefinedVarsTests.cs index a04c77853..dec8c849a 100644 --- a/src/Analysis/Ast/Test/LintUndefinedVarsTests.cs +++ b/src/Analysis/Ast/Test/LintUndefinedVarsTests.cs @@ -764,7 +764,7 @@ public async Task NamedExpr() { if (y := x): print(y) "; - var d = await LintAsync(code, PythonVersions.Python38_x64); + var d = await LintAsync(code, PythonVersions.Required_Python38X); d.Should().BeEmpty(); } @@ -777,10 +777,62 @@ public async Task NamedExprInComprehension() { [i+1 for i in range(2) for j in (k := stuff)] [i+1 for i in [j for j in (k := stuff)]] "; - var d = await LintAsync(code, PythonVersions.Python38_x64); + var d = await LintAsync(code, PythonVersions.Required_Python38X); d.Should().BeEmpty(); } + [TestMethod, Priority(0)] + public async Task UnresolvedFromImport() { + const string code = @" +from thismoduledoesnotexist import something +something() +"; + var d = await LintAsync(code); + d.Should().BeEmpty(); + } + + [TestMethod, Priority(0)] + public async Task AssignedSelf() { + const string code = @" +class DocEnum(Enum): + def __new__(cls, value, doc=None): + self = object.__new__(cls) + self._value_ = value + if doc is not None: + self.__doc__ = doc + return self +"; + var d = await LintAsync(code); + d.Should().BeEmpty(); + } + + [TestMethod, Priority(0)] + public async Task Metaclass() { + const string code = @" +class MyMetaclass(type): + pass + +MyClass = MyMetaclass('MyClass', (), {}) + +class Subclass(MyClass): + pass +"; + var d = await LintAsync(code); + d.Should().BeEmpty(); + } + + [TestMethod, Priority(0)] + public async Task AugmentedAssignToUndefined() { + const string code = @" +x += 1 +"; + var d = await LintAsync(code); + d.Should().HaveCount(1); + d[0].ErrorCode.Should().Be(ErrorCodes.UndefinedVariable); + d[0].SourceSpan.Should().Be(2, 1, 2, 2); + } + + private async Task> LintAsync(string code, InterpreterConfiguration configuration = null) { var analysis = await GetAnalysisAsync(code, configuration ?? PythonVersions.LatestAvailable3X); var a = Services.GetService(); diff --git a/src/Analysis/Ast/Test/Microsoft.Python.Analysis.Tests.csproj b/src/Analysis/Ast/Test/Microsoft.Python.Analysis.Tests.csproj index 0797b6812..ac8babced 100644 --- a/src/Analysis/Ast/Test/Microsoft.Python.Analysis.Tests.csproj +++ b/src/Analysis/Ast/Test/Microsoft.Python.Analysis.Tests.csproj @@ -1,6 +1,6 @@  - netcoreapp3.0 + netcoreapp3.1 Microsoft.Python.Analysis.Tests Microsoft.Python.Analysis.Tests @@ -23,15 +23,15 @@ - - - - + + + + all runtime; build; native; contentfiles; analyzers - + diff --git a/src/Analysis/Ast/Test/ReferencesTests.cs b/src/Analysis/Ast/Test/ReferencesTests.cs index bf1fa2418..5e367b462 100644 --- a/src/Analysis/Ast/Test/ReferencesTests.cs +++ b/src/Analysis/Ast/Test/ReferencesTests.cs @@ -13,7 +13,6 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. -using System.ComponentModel.DataAnnotations; using System.Threading.Tasks; using FluentAssertions; using Microsoft.Python.Analysis.Tests.FluentAssertions; @@ -441,5 +440,76 @@ public async Task ExtendAllAssignment() { all.References[3].Span.Should().Be(12, 1, 12, 8); all.References[4].Span.Should().Be(13, 1, 13, 8); } + + [TestMethod, Priority(0)] + public async Task VariableInCallParameters() { + const string code = @" +from constants import * +import constants + +print(VARIABLE1) +print(constants.VARIABLE1) +x = print(VARIABLE1) +"; + await TestData.CreateTestSpecificFileAsync("constants.py", @"VARIABLE1 = 'afad'"); + var analysis = await GetAnalysisAsync(code); + var v1 = analysis.Should().HaveVariable("VARIABLE1").Which; + + v1.Definition.Span.Should().Be(1, 1, 1, 10); + v1.Definition.DocumentUri.AbsolutePath.Should().Contain("constants.py"); + + v1.References.Should().HaveCount(4); + v1.References[0].Span.Should().Be(1, 1, 1, 10); + v1.References[0].DocumentUri.AbsolutePath.Should().Contain("constants.py"); + + v1.References[1].Span.Should().Be(5, 7, 5, 16); + v1.References[1].DocumentUri.AbsolutePath.Should().Contain("module.py"); + + v1.References[2].Span.Should().Be(6, 17, 6, 26); + v1.References[2].DocumentUri.AbsolutePath.Should().Contain("module.py"); + + v1.References[3].Span.Should().Be(7, 11, 7, 20); + v1.References[3].DocumentUri.AbsolutePath.Should().Contain("module.py"); + } + + [TestMethod, Priority(0)] + public async Task LibraryFunction() { + const string code = @" +print(1) +print(2) +"; + var analysis = await GetAnalysisAsync(code); + var b = analysis.Document.Interpreter.ModuleResolution.BuiltinsModule; + var print = b.Analysis.Should().HaveVariable("print").Which; + + print.Definition.Span.Should().Be(1, 1, 1, 1); + + print.References.Should().HaveCount(3); + print.References[0].Span.Should().Be(1, 1, 1, 1); + print.References[0].DocumentUri.AbsolutePath.Should().Contain("python.pyi"); + + print.References[1].Span.Should().Be(2, 1, 2, 6); + print.References[1].DocumentUri.AbsolutePath.Should().Contain("module.py"); + + print.References[2].Span.Should().Be(3, 1, 3, 6); + print.References[2].DocumentUri.AbsolutePath.Should().Contain("module.py"); + } + + [TestMethod, Priority(0)] + public async Task GlobalInSequence() { + const string code = @" +LINE_NO = 0 + +def funky(filename): + global LINE_NO + with open(filename, 'r') as file_desc: + LINE_NO = 0 + for line in file_desc: + other, LINE_NO = process(line) +"; + var analysis = await GetAnalysisAsync(code); + var v = analysis.Should().HaveVariable("LINE_NO").Which; + v.Definition.StartLine.Should().Be(2); + } } } diff --git a/src/Analysis/Ast/Test/ScrapeTests.cs b/src/Analysis/Ast/Test/ScrapeTests.cs index 34d6a8caf..d3278475d 100644 --- a/src/Analysis/Ast/Test/ScrapeTests.cs +++ b/src/Analysis/Ast/Test/ScrapeTests.cs @@ -272,8 +272,9 @@ private async Task FullStdLibTest(InterpreterConfiguration configuration, params var interpreter = services.GetService(); var pathResolver = interpreter.ModuleResolution.CurrentPathResolver; - var modules = pathResolver.GetAllModuleNames() + var modules = pathResolver.GetAllImportableModuleNames() .Select(n => pathResolver.GetModuleImportFromModuleName(n)) + .ExcludeDefault() .Where(i => i.RootPath.PathEquals(configuration.SitePackagesPath) || i.RootPath.PathEquals(configuration.LibraryPath)) .ToList(); diff --git a/src/Analysis/Ast/Test/TypingTests.cs b/src/Analysis/Ast/Test/TypingTests.cs index 31d9fd1de..3a8a72396 100644 --- a/src/Analysis/Ast/Test/TypingTests.cs +++ b/src/Analysis/Ast/Test/TypingTests.cs @@ -329,6 +329,52 @@ def m1(self): ... .Which.Should().HaveType(BuiltinTypeId.Str); } + [TestMethod, Priority(0)] + public async Task TupleOfNones() { + const string code = @" +from typing import Tuple + +x: Tuple[None, None, None] +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); + + analysis.Should().HaveVariable("x") + .Which.Should().HaveType("Tuple[None, None, None]"); + } + + [TestMethod, Priority(0)] + public async Task UnionOfTuples() { + const string code = @" +from typing import Union, Tuple, Type + +class TracebackType: ... + +_ExcInfo = Tuple[Type[BaseException], BaseException, TracebackType] +_OptExcInfo = Union[_ExcInfo, Tuple[None, None, None]] + +x: _ExcInfo +y: _OptExcInfo + +a, b, c = y +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); + + analysis.Should().HaveVariable("x") + .Which.Should().HaveType("Tuple[Type[BaseException], BaseException, TracebackType]"); + + analysis.Should().HaveVariable("y") + .Which.Should().HaveType("Union[Tuple[Type[BaseException], BaseException, TracebackType], Tuple[None, None, None]]"); + + analysis.Should().HaveVariable("a") + .Which.Should().HaveType("Union[Type[BaseException], None]"); + + analysis.Should().HaveVariable("b") + .Which.Should().HaveType("Union[BaseException, None]"); + + analysis.Should().HaveVariable("c") + .Which.Should().HaveType("Union[TracebackType, None]"); + } + [TestMethod, Priority(0)] public void AnnotationParsing() { AssertTransform("List", "NameOp:List"); @@ -340,6 +386,34 @@ public void AnnotationParsing() { AssertTransform("Dict['Int, Str']", "NameOp:Dict", "StartListOp", "NameOp:Int", "NameOp:Str", "MakeGenericOp"); } + [TestMethod, Priority(0)] + public async Task UnionsCompare() { + var analysis = await GetAnalysisAsync("from typing import Union", PythonVersions.LatestAvailable3X); + var i = analysis.Document.Interpreter.GetBuiltinType(BuiltinTypeId.Int); + var b = analysis.Document.Interpreter.GetBuiltinType(BuiltinTypeId.Bool); + + var expected = new[] {i, b}; + var u1 = new PythonUnionType(expected, analysis.Document); + var u2 = new PythonUnionType(expected, analysis.Document); + + u2.Equals(u1).Should().BeTrue(); + u1.Add(u2); + var actual = u1.ToArray(); + actual.Should().Equal(expected); + + u1.Add(u1); + actual = u1.ToArray(); + actual.Should().Equal(expected); + + u2.Add(u1); + actual = u2.ToArray(); + actual.Should().Equal(expected); + + u2.Add(u2); + actual = u1.ToArray(); + actual.Should().Equal(expected); + } + [TestMethod, Priority(0)] public void AnnotationConversion() { AssertConvert("List"); diff --git a/src/Analysis/Core/Impl/DependencyResolution/AstUtilities.cs b/src/Analysis/Core/Impl/DependencyResolution/AstUtilities.cs index 57c93cc1d..8c7ee1457 100644 --- a/src/Analysis/Core/Impl/DependencyResolution/AstUtilities.cs +++ b/src/Analysis/Core/Impl/DependencyResolution/AstUtilities.cs @@ -13,9 +13,8 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. -using System.Collections.Generic; -using System.Linq; using Microsoft.Python.Parsing.Ast; +using System.Collections.Generic; namespace Microsoft.Python.Analysis.Core.DependencyResolution { public static class AstUtilities { diff --git a/src/Analysis/Core/Impl/DependencyResolution/PathResolverSnapshot.cs b/src/Analysis/Core/Impl/DependencyResolution/PathResolverSnapshot.cs index 0ccf32906..faa91140a 100644 --- a/src/Analysis/Core/Impl/DependencyResolution/PathResolverSnapshot.cs +++ b/src/Analysis/Core/Impl/DependencyResolution/PathResolverSnapshot.cs @@ -46,19 +46,20 @@ public partial class PathResolverSnapshot { private readonly string _workDirectory; private readonly ImmutableArray _interpreterSearchPaths; private readonly ImmutableArray _userSearchPaths; + private readonly HashSet _isLibraryPath; private readonly ImmutableArray _roots; private readonly int _userRootsCount; public int Version { get; } public PathResolverSnapshot(PythonLanguageVersion pythonLanguageVersion, string workDirectory, ImmutableArray interpreterSearchPaths, ImmutableArray userSearchPaths) - : this(pythonLanguageVersion, workDirectory, userSearchPaths, interpreterSearchPaths, ImmutableArray.Empty, 0, Node.CreateDefaultRoot(), Node.CreateBuiltinRoot(), default) { + : this(pythonLanguageVersion, workDirectory, userSearchPaths, interpreterSearchPaths, CreateIsLibrarySet(userSearchPaths, interpreterSearchPaths), ImmutableArray.Empty, 0, Node.CreateDefaultRoot(), Node.CreateBuiltinRoot(), default) { _pythonLanguageVersion = pythonLanguageVersion; _workDirectory = workDirectory; _userSearchPaths = userSearchPaths; _interpreterSearchPaths = interpreterSearchPaths; - if (workDirectory != string.Empty) { + if (!string.IsNullOrEmpty(workDirectory)) { CreateRootsWithDefault(workDirectory, userSearchPaths, interpreterSearchPaths, out _roots, out _userRootsCount); } else { CreateRootsWithoutDefault(userSearchPaths, interpreterSearchPaths, out _roots, out _userRootsCount); @@ -69,11 +70,12 @@ public PathResolverSnapshot(PythonLanguageVersion pythonLanguageVersion, string Version = default; } - private PathResolverSnapshot(PythonLanguageVersion pythonLanguageVersion, string workDirectory, ImmutableArray userSearchPaths, ImmutableArray interpreterSearchPaths, ImmutableArray roots, int userRootsCount, Node nonRooted, Node builtins, int version) { + private PathResolverSnapshot(PythonLanguageVersion pythonLanguageVersion, string workDirectory, ImmutableArray userSearchPaths, ImmutableArray interpreterSearchPaths, HashSet isLibraryPath, ImmutableArray roots, int userRootsCount, Node nonRooted, Node builtins, int version) { _pythonLanguageVersion = pythonLanguageVersion; _workDirectory = workDirectory; _userSearchPaths = userSearchPaths; _interpreterSearchPaths = interpreterSearchPaths; + _isLibraryPath = isLibraryPath; _roots = roots; _userRootsCount = userRootsCount; _nonRooted = nonRooted; @@ -81,28 +83,45 @@ private PathResolverSnapshot(PythonLanguageVersion pythonLanguageVersion, string Version = version; } - public ImmutableArray GetAllModuleNames() => GetModuleNames(_roots.Prepend(_nonRooted)); - public IEnumerable GetInterpreterModuleNames() => GetModuleNames(_roots.Skip(_userRootsCount).Append(_builtins)); + public ImmutableArray GetAllImportableModuleNames(bool includeImplicitPackages = true) { + return GetAllImportableModuleInfo(n => !string.IsNullOrEmpty(n.FullModuleName), n => n.FullModuleName, includeImplicitPackages); + } + + public ImmutableArray GetAllImportableModulesByName(string name, bool includeImplicitPackages = true) { + return GetAllImportableModuleInfo(n => string.Equals(n.Name, name), n => n.FullModuleName, includeImplicitPackages); + } - private ImmutableArray GetModuleNames(IEnumerable roots) { + public ImmutableArray GetAllImportableModuleFilePaths(bool includeImplicitPackages = true) { + return GetAllImportableModuleInfo(n => !string.IsNullOrEmpty(n.ModulePath), n => n.ModulePath, includeImplicitPackages); + } + + private ImmutableArray GetAllImportableModuleInfo(Func predicate, Func valueGetter, bool includeImplicitPackages = true) { + var roots = _roots.Prepend(_nonRooted); var items = new Queue(roots); - var names = ImmutableArray.Empty; + var stringValues = ImmutableArray.Empty; + while (items.Count > 0) { var item = items.Dequeue(); - if (item.IsModule) { - names = names.Add(item.FullModuleName); - } else { - foreach (var child in item.Children) { + if (item != null) { + if (predicate(item) && (item.IsModule || includeImplicitPackages)) { + stringValues = stringValues.Add(valueGetter(item)); + } + + foreach (var child in item.Children.ExcludeDefault()) { items.Enqueue(child); } } } - foreach (var builtin in _builtins.Children) { - names = names.Add(builtin.FullModuleName); - } + return stringValues.AddRange( + _builtins.Children + .Where(b => predicate(b)) + .Select(b => valueGetter(b)) + ); + } - return names; + public string GetModuleNameByPath(string modulePath) { + return TryFindModule(modulePath, out var edge, out _) ? edge.End.FullModuleName : null; } public ModuleImport GetModuleImportFromModuleName(in string fullModuleName) { @@ -394,6 +413,7 @@ public PathResolverSnapshot SetBuiltins(in IEnumerable builtinModuleName _workDirectory, _userSearchPaths, _interpreterSearchPaths, + _isLibraryPath, _roots, _userRootsCount, _nonRooted, @@ -708,10 +728,10 @@ private static void AppendNameIfNotInitPy(StringBuilder builder, string name) { } private static void AppendName(StringBuilder builder, string name) - => builder.AppendIf(builder.Length > 0, ".").Append(name); + => builder.AppendIf(builder.Length > 0 && builder[builder.Length - 1] != '.', ".").Append(name); private static Node UpdateNodesFromEnd(Edge lastEdge, Node newEnd) { - while (lastEdge.Start != default) { + while (lastEdge.Start != null) { var newStart = lastEdge.Start.ReplaceChildAt(newEnd, lastEdge.EndIndex); lastEdge = lastEdge.Previous; newEnd = newStart; @@ -798,14 +818,17 @@ private static bool IsNotInitPy(in Node node) private static bool IsInitPyModule(in Node node, out Node initPyNode) => node.TryGetChild("__init__", out initPyNode) && initPyNode.IsModule; - private bool IsLibraryPath(string rootPath) - => !_userSearchPaths.Contains(rootPath, StringExtensions.PathsStringComparer) - && _interpreterSearchPaths.Except(_userSearchPaths).Contains(rootPath, StringExtensions.PathsStringComparer); + private bool IsLibraryPath(string rootPath) => _isLibraryPath.Contains(rootPath); private PathResolverSnapshot ReplaceNonRooted(Node nonRooted) - => new PathResolverSnapshot(_pythonLanguageVersion, _workDirectory, _userSearchPaths, _interpreterSearchPaths, _roots, _userRootsCount, nonRooted, _builtins, Version + 1); + => new PathResolverSnapshot(_pythonLanguageVersion, _workDirectory, _userSearchPaths, _interpreterSearchPaths, _isLibraryPath, _roots, _userRootsCount, nonRooted, _builtins, Version + 1); private PathResolverSnapshot ImmutableReplaceRoot(Node root, int index) - => new PathResolverSnapshot(_pythonLanguageVersion, _workDirectory, _userSearchPaths, _interpreterSearchPaths, _roots.ReplaceAt(index, root), _userRootsCount, _nonRooted, _builtins, Version + 1); + => new PathResolverSnapshot(_pythonLanguageVersion, _workDirectory, _userSearchPaths, _interpreterSearchPaths, _isLibraryPath, _roots.ReplaceAt(index, root), _userRootsCount, _nonRooted, _builtins, Version + 1); + + private static HashSet CreateIsLibrarySet(ImmutableArray userSearchPaths, ImmutableArray interpreterSearchPaths) { + var interpreterMinusUser = interpreterSearchPaths.Except(userSearchPaths, StringExtensions.PathsStringComparer); + return new HashSet(interpreterMinusUser, StringExtensions.PathsStringComparer); + } } } diff --git a/src/Analysis/Core/Impl/Interpreter/InterpreterConfiguration.cs b/src/Analysis/Core/Impl/Interpreter/InterpreterConfiguration.cs index b500f1e9d..4e4f58a04 100644 --- a/src/Analysis/Core/Impl/Interpreter/InterpreterConfiguration.cs +++ b/src/Analysis/Core/Impl/Interpreter/InterpreterConfiguration.cs @@ -31,8 +31,7 @@ internal InterpreterConfiguration( string libPath = null, string sitePackagesPath = null, InterpreterArchitecture architecture = default, - Version version = null - ) { + Version version = null) { InterpreterPath = interpreterPath; PathEnvironmentVariable = pathVar; Architecture = architecture ?? InterpreterArchitecture.Unknown; diff --git a/src/Analysis/Core/Impl/Interpreter/PythonLibraryPath.cs b/src/Analysis/Core/Impl/Interpreter/PythonLibraryPath.cs index 671845dd0..d51d60087 100644 --- a/src/Analysis/Core/Impl/Interpreter/PythonLibraryPath.cs +++ b/src/Analysis/Core/Impl/Interpreter/PythonLibraryPath.cs @@ -174,7 +174,7 @@ public static async Task> GetSearchPathsFromIn var output = await ps.ExecuteAndCaptureOutputAsync(startInfo, cancellationToken); return output.Split(new[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries) - .Skip(1).Select(s => { + .Select(s => { try { return Parse(s); } catch (ArgumentException) { diff --git a/src/Analysis/Core/Impl/Microsoft.Python.Analysis.Core.csproj b/src/Analysis/Core/Impl/Microsoft.Python.Analysis.Core.csproj index 4e97b8e58..99cc45a00 100644 --- a/src/Analysis/Core/Impl/Microsoft.Python.Analysis.Core.csproj +++ b/src/Analysis/Core/Impl/Microsoft.Python.Analysis.Core.csproj @@ -9,7 +9,7 @@ 1701, 1702 - "You may need to supply assembly policy" --> 1701;1702;$(NoWarn) - 7.2 + true diff --git a/src/Caching/Impl/IO/CacheWriter.cs b/src/Caching/Impl/IO/CacheWriter.cs new file mode 100644 index 000000000..269fba7ae --- /dev/null +++ b/src/Caching/Impl/IO/CacheWriter.cs @@ -0,0 +1,93 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using LiteDB; +using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Core; +using Microsoft.Python.Core.IO; +using Microsoft.Python.Core.Logging; +using Microsoft.Python.Core.Threading; + +namespace Microsoft.Python.Analysis.Caching.IO { + internal sealed class CacheWriter : IDisposable { + private readonly IFileSystem _fs; + private readonly ILogger _log; + private readonly string _cacheFolder; + private readonly TaskQueue _taskQueue; + private readonly IPythonAnalyzer _analyzer; + + public CacheWriter(IPythonAnalyzer analyzer, IFileSystem fs, ILogger log, string cacheFolder) { + _fs = fs; + _log = log; + _cacheFolder = cacheFolder; + _taskQueue = new TaskQueue(Math.Max(1, Environment.ProcessorCount / 4)); + + _analyzer = analyzer; + _analyzer.AnalysisComplete += OnAnalysisComplete; + } + + private void OnAnalysisComplete(object sender, AnalysisCompleteEventArgs e) => _taskQueue.ProcessQueue(); + + public void Dispose() { + _analyzer.AnalysisComplete -= OnAnalysisComplete; + _taskQueue.Dispose(); + } + + public Task EnqueueModel(ModuleModel model, bool immediate = false, CancellationToken cancellationToken = default) { + var tcs = new TaskCompletionSource(); + _taskQueue.Enqueue(() => { + try { + Write(model, cancellationToken); + tcs.SetResult(true); + } catch (OperationCanceledException) { + tcs.TrySetCanceled(); + } catch (Exception ex) when (!ex.IsCriticalException()) { + tcs.TrySetException(ex); + } + }, immediate: immediate); + return tcs.Task; + } + + private void Write(ModuleModel model, CancellationToken cancellationToken) { + if (cancellationToken.IsCancellationRequested) { + return; + } + + WithRetries.Execute(() => { + if (!_fs.DirectoryExists(_cacheFolder)) { + _fs.CreateDirectory(_cacheFolder); + } + return true; + }, $"Unable to create directory {_cacheFolder} for modules cache.", _log); + + WithRetries.Execute(() => { + if (cancellationToken.IsCancellationRequested) { + return false; + } + using (var db = new LiteDatabase(Path.Combine(_cacheFolder, $"{model.UniqueId}.db"))) { + var modules = db.GetCollection("modules"); + modules.Upsert(model); + modules.EnsureIndex(x => x.Name); + } + return true; + }, $"Unable to write analysis of {model.Name} to database.", _log); + } + } +} diff --git a/src/Caching/Impl/Lazy/MemberFactory.cs b/src/Caching/Impl/Lazy/MemberFactory.cs new file mode 100644 index 000000000..31e6ebdcf --- /dev/null +++ b/src/Caching/Impl/Lazy/MemberFactory.cs @@ -0,0 +1,53 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Diagnostics; +using System.Linq; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Specializations.Typing.Types; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; + +namespace Microsoft.Python.Analysis.Caching.Lazy { + internal static class MemberFactory { + public static IMember CreateMember(MemberModel model, ModuleFactory mf, IGlobalScope gs, IPythonType declaringType) { + var unkType = mf.Module.Interpreter.UnknownType; + switch (model) { + case ClassModel cm: + return new PythonLazyClassType(cm, mf, gs, declaringType); + case FunctionModel fm: + return new PythonLazyFunctionType(fm, mf, gs, declaringType); + case PropertyModel pm: + return new PythonLazyPropertyType(pm, mf, gs, declaringType); + + case NamedTupleModel ntm: + var itemTypes = ntm.ItemTypes.Select(n => mf.ConstructType(n) ?? unkType).ToArray(); + return new NamedTupleType(ntm.Name, ntm.ItemNames, itemTypes, mf.Module, ntm.IndexSpan.ToSpan()); + + case TypeVarModel tvm: + return new GenericTypeParameter(tvm.Name, mf.Module, + tvm.Constraints.Select(n => mf.ConstructType(n) ?? unkType).ToArray(), + mf.ConstructType(tvm.Bound), tvm.Covariant, tvm.Contravariant, default); + + case VariableModel vm: + var m = mf.ConstructMember(vm.Value) ?? unkType; + return new Variable(vm.Name, m, VariableSource.Declaration, new Location(mf.Module, vm.IndexSpan?.ToSpan() ?? default)); + + } + Debug.Fail("Unsupported model type."); + return null; + } + } +} diff --git a/src/Caching/Impl/Lazy/PythonLazyClassType.cs b/src/Caching/Impl/Lazy/PythonLazyClassType.cs new file mode 100644 index 000000000..27f9a612b --- /dev/null +++ b/src/Caching/Impl/Lazy/PythonLazyClassType.cs @@ -0,0 +1,147 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Specializations.Typing; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Types.Collections; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core; +using Microsoft.Python.Parsing; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Caching.Lazy { + internal sealed class PythonLazyClassType : PythonLazyType, IPythonClassType { + private readonly PythonClassType _cls; + + public PythonLazyClassType(ClassModel model, ModuleFactory mf, IGlobalScope gs, IPythonType declaringType) + : base(model, mf, gs, declaringType) { + _cls = new PythonClassType(model.Name, new Location(mf.Module, model.IndexSpan.ToSpan())); + _cls.SetDocumentation(model.Documentation); + SetInnerType(_cls); + } + + #region IPythonClassType + public IPythonType CreateSpecificType(IArgumentSet typeArguments) { + EnsureContent(); + return _cls.CreateSpecificType(typeArguments); + } + + public IReadOnlyList Parameters { + get { + EnsureContent(); + return _cls.Parameters; + } + } + + public bool IsGeneric { + get { + EnsureContent(); + return _cls.IsGeneric; + } + } + + public ClassDefinition ClassDefinition => null; + + public IReadOnlyList Mro { + get { + EnsureContent(); + return _cls.Mro; + } + } + + public IReadOnlyList Bases { + get { + EnsureContent(); + return _cls.Bases; + } + } + + public IReadOnlyDictionary GenericParameters { + get { + EnsureContent(); + return _cls.GenericParameters; + } + } + #endregion + + protected override void EnsureContent(ClassModel cm) { + var bases = CreateBases(cm, ModuleFactory, GlobalScope); + _cls.SetBases(bases); + + if (cm.GenericParameterValues.Length > 0) { + _cls.StoreGenericParameters( + _cls, + _cls.GenericParameters.Keys.ToArray(), + cm.GenericParameterValues.ToDictionary( + k => _cls.GenericParameters.Keys.First(x => x == k.Name), + v => ModuleFactory.ConstructType(v.Type) + ) + ); + } + + foreach (var model in GetMemberModels(cm)) { + _cls.AddMember(model.Name, MemberFactory.CreateMember(model, ModuleFactory, GlobalScope, _cls), false); + } + _cls.AddMember("__class__", _cls, true); + } + + private IEnumerable CreateBases(ClassModel cm, ModuleFactory mf, IGlobalScope gs) { + var ntBases = cm.NamedTupleBases + .Select(ntb => MemberFactory.CreateMember(ntb, ModuleFactory, GlobalScope, _cls)) + .OfType() + .ToArray(); + + var is3x = mf.Module.Interpreter.LanguageVersion.Is3x(); + var basesNames = cm.Bases.Select(b => is3x && b == "object" ? null : b).ExcludeDefault().ToArray(); + var bases = basesNames.Select(mf.ConstructType).ExcludeDefault().Concat(ntBases).ToArray(); + + // Make sure base types are realized + foreach (var b in bases.OfType()) { + b.EnsureContent(); + } + + if (cm.GenericBaseParameters.Length > 0) { + // Generic class. Need to reconstruct generic base so code can then + // create specific types off the generic class. + var genericBase = bases.OfType().FirstOrDefault(b => b.Name == "Generic"); + if (genericBase != null) { + var typeVars = cm.GenericBaseParameters.Select(n => gs.Variables[n]?.Value).OfType().ToArray(); + //Debug.Assert(typeVars.Length > 0, "Class generic type parameters were not defined in the module during restore"); + if (typeVars.Length > 0) { + var genericWithParameters = genericBase.CreateSpecificType(new ArgumentSet(typeVars, null, null)); + if (genericWithParameters != null) { + bases = bases.Except(Enumerable.Repeat(genericBase, 1)).Concat(Enumerable.Repeat(genericWithParameters, 1)).ToArray(); + } + } + } else { + Debug.Fail("Generic class does not have generic base."); + } + } + + if (bases.Length > 0) { + _cls.AddMember("__base__", bases[0], true); + } + _cls.AddMember("__bases__", PythonCollectionType.CreateList(DeclaringModule.Interpreter.ModuleResolution.BuiltinsModule, bases), true); + return bases; + } + + protected override IEnumerable GetMemberModels(ClassModel cm) + => cm.Classes.Concat(cm.Properties).Concat(cm.Methods).Concat(cm.Fields); + } +} diff --git a/src/Caching/Impl/Lazy/PythonLazyFunctionType.cs b/src/Caching/Impl/Lazy/PythonLazyFunctionType.cs new file mode 100644 index 000000000..527c45b03 --- /dev/null +++ b/src/Caching/Impl/Lazy/PythonLazyFunctionType.cs @@ -0,0 +1,60 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Caching.Lazy { + internal sealed class PythonLazyFunctionType : PythonLazyType, IPythonFunctionType { + private readonly PythonFunctionType _function; + + public PythonLazyFunctionType(FunctionModel model, ModuleFactory mf, IGlobalScope gs, IPythonType declaringType) + : base(model, mf, gs, declaringType) { + var location = new Location(mf.Module, model.IndexSpan.ToSpan()); + _function = new PythonFunctionType(model.Name, location, declaringType, model.Documentation); + + // TODO: restore signature string so hover (tooltip) documentation won't have to restore the function. + // parameters and return type just to look at them. + foreach (var om in model.Overloads) { + var o = new PythonLazyOverload(om, mf, _function); + _function.AddOverload(o); + } + SetInnerType(_function); + } + + #region IPythonFunctionType + public FunctionDefinition FunctionDefinition => null; + public bool IsClassMethod => _function.IsClassMethod; + public bool IsStatic => _function.IsStatic; + public bool IsOverload => _function.IsStatic; + public bool IsStub => _function.IsStatic; + public bool IsUnbound => _function.IsStatic; + public IReadOnlyList Overloads => _function.Overloads; + #endregion + + protected override void EnsureContent(FunctionModel fm) { + foreach (var model in GetMemberModels(fm)) { + _function.AddMember(model.Name, MemberFactory.CreateMember(model, ModuleFactory, GlobalScope, _function), overwrite: true); + } + } + + protected override IEnumerable GetMemberModels(FunctionModel fm) + => fm.Classes.Concat(fm.Functions); + } +} diff --git a/src/Caching/Impl/Lazy/PythonLazyOverload.cs b/src/Caching/Impl/Lazy/PythonLazyOverload.cs new file mode 100644 index 000000000..edce21be7 --- /dev/null +++ b/src/Caching/Impl/Lazy/PythonLazyOverload.cs @@ -0,0 +1,83 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Caching.Lazy { + internal sealed class PythonLazyOverload: IPythonFunctionOverload { + private readonly object _contentLock = new object(); + private readonly PythonFunctionOverload _overload; + private ModuleFactory _mf; + private OverloadModel _model; + + public PythonLazyOverload(OverloadModel model, ModuleFactory mf, IPythonClassMember cm) { + _model = model; + _mf = mf; + + ClassMember = cm; + Documentation = model.Documentation; + + _overload = new PythonFunctionOverload(cm, new Location(mf.Module, default)); + _overload.SetDocumentation(cm.Documentation); + } + + public FunctionDefinition FunctionDefinition => null; + public IPythonClassMember ClassMember { get; } + public string Name => ClassMember.Name; + public string Documentation { get; } + + public IReadOnlyList Parameters { + get { + EnsureContent(); + return _overload.Parameters; + } + } + + public IMember Call(IArgumentSet args, IPythonType self) { + EnsureContent(); + return _overload.Call(args, self); + } + + public string GetReturnDocumentation(IPythonType self = null) { + EnsureContent(); + return _overload.GetReturnDocumentation(self); + } + + public IMember StaticReturnValue { + get { + EnsureContent(); + return _overload.StaticReturnValue; + } + } + + private void EnsureContent() { + lock (_contentLock) { + if (_model != null) { + _overload.SetParameters(_model.Parameters.Select(p => ConstructParameter(_mf, p)).ToArray()); + _overload.SetReturnValue(_mf.ConstructMember(_model.ReturnType), true); + _model = null; + _mf = null; + } + } + } + + private IParameterInfo ConstructParameter(ModuleFactory mf, ParameterModel pm) + => new ParameterInfo(pm.Name, mf.ConstructType(pm.Type), pm.Kind, mf.ConstructMember(pm.DefaultValue)); + } +} diff --git a/src/Caching/Impl/Lazy/PythonLazyPropertyType.cs b/src/Caching/Impl/Lazy/PythonLazyPropertyType.cs new file mode 100644 index 000000000..47d605ab8 --- /dev/null +++ b/src/Caching/Impl/Lazy/PythonLazyPropertyType.cs @@ -0,0 +1,59 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Analysis.Caching.Lazy { + internal sealed class PythonLazyPropertyType : PythonLazyType, IPythonPropertyType { + private readonly PythonPropertyType _property; + + public PythonLazyPropertyType(PropertyModel model, ModuleFactory mf, IGlobalScope gs, IPythonType declaringType) + : base(model, mf, gs, declaringType) { + + var location = new Location(mf.Module, model.IndexSpan.ToSpan()); + _property = new PythonPropertyType(model.Name, location, model.Documentation, declaringType, + model.Attributes.HasFlag(FunctionAttributes.Abstract)); + + // parameters and return type just to look at them. + var o = new PythonFunctionOverload(_property, location); + o.SetDocumentation(model.Documentation); + _property.AddOverload(o); + + IsReadOnly = model.IsReadOnly; + SetInnerType(_property); + } + + public FunctionDefinition FunctionDefinition => null; + public bool IsReadOnly { get; } + + public IMember ReturnType { + get { + EnsureContent(); + return _property.ReturnType; + } + } + + protected override void EnsureContent(PropertyModel pm) { + _property.Getter.SetReturnValue(ModuleFactory.ConstructMember(pm.ReturnType), true); + } + protected override IEnumerable GetMemberModels(PropertyModel pm) + => pm.Classes.Concat(pm.Functions); + } +} diff --git a/src/Caching/Impl/Lazy/PythonLazyType.cs b/src/Caching/Impl/Lazy/PythonLazyType.cs new file mode 100644 index 000000000..421fbf9fc --- /dev/null +++ b/src/Caching/Impl/Lazy/PythonLazyType.cs @@ -0,0 +1,75 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core; + +namespace Microsoft.Python.Analysis.Caching.Lazy { + /// + /// Represents 'lazy' type that delays creation of its content such as members, + /// function parameters and return types until they are requested. This allows + /// deferred fetching of data from the database, avoiding wholesale restore. + /// + internal abstract class PythonLazyType : PythonTypeWrapper where TModel : class { + private readonly object _contentLock = new object(); + private TModel _model; + + protected IGlobalScope GlobalScope { get; private set; } + protected ModuleFactory ModuleFactory { get; private set; } + + protected PythonLazyType(TModel model, ModuleFactory mf, IGlobalScope gs, IPythonType declaringType) { + _model = model ?? throw new ArgumentNullException(nameof(model)); + ModuleFactory = mf ?? throw new ArgumentNullException(nameof(mf)); + GlobalScope = gs ?? throw new ArgumentNullException(nameof(gs)); + DeclaringType = declaringType; + } + + public IPythonType DeclaringType { get; } + + #region IPythonType + + public override IMember GetMember(string name) { + if (_model != null) { + var memberModel = GetMemberModels(_model).FirstOrDefault(m => m.Name == name); + return memberModel != null ? MemberFactory.CreateMember(memberModel, ModuleFactory, GlobalScope, this) : null; + } + return base.GetMember(name); + } + + public override IEnumerable GetMemberNames() + => _model != null ? GetMemberModels(_model).Select(m => m.Name) : base.GetMemberNames(); + #endregion + + internal void EnsureContent() { + lock (_contentLock) { + if (_model != null) { + EnsureContent(_model); + + ModuleFactory = null; + GlobalScope = null; + _model = null; + } + } + } + + protected abstract void EnsureContent(TModel model); + protected abstract IEnumerable GetMemberModels(TModel m); + } +} diff --git a/src/Caching/Impl/Microsoft.Python.Analysis.Caching.csproj b/src/Caching/Impl/Microsoft.Python.Analysis.Caching.csproj index 73ddbba31..8f592a7c3 100644 --- a/src/Caching/Impl/Microsoft.Python.Analysis.Caching.csproj +++ b/src/Caching/Impl/Microsoft.Python.Analysis.Caching.csproj @@ -1,6 +1,6 @@  - netstandard2.1 + netstandard2.0 Microsoft.Python.Analysis.Caching Microsoft.Python.Analysis.Caching @@ -9,7 +9,7 @@ 1701, 1702 - "You may need to supply assembly policy" --> 1701;1702;$(NoWarn) - 7.2 + true diff --git a/src/Caching/Impl/Models/ModuleModel.cs b/src/Caching/Impl/Models/ModuleModel.cs index fbca1705f..8adf55a86 100644 --- a/src/Caching/Impl/Models/ModuleModel.cs +++ b/src/Caching/Impl/Models/ModuleModel.cs @@ -38,7 +38,6 @@ internal sealed class ModuleModel : MemberModel { public ClassModel[] Classes { get; set; } public TypeVarModel[] TypeVars { get; set; } public NamedTupleModel[] NamedTuples { get; set; } - /// /// Collection of new line information for conversion of linear spans /// to line/columns in navigation to member definitions and references. @@ -69,7 +68,6 @@ public static ModuleModel FromAnalysis(IDocumentAnalysis analysis, IServiceConta var classes = new Dictionary(); var typeVars = new Dictionary(); var namedTuples = new Dictionary(); - // Go directly through variables which names are listed in GetMemberNames // as well as variables that are declarations. var exportedNames = new HashSet(analysis.Document.GetMemberNames()); diff --git a/src/Caching/Impl/Models/SubmoduleModel.cs b/src/Caching/Impl/Models/SubmoduleModel.cs new file mode 100644 index 000000000..3b751adc9 --- /dev/null +++ b/src/Caching/Impl/Models/SubmoduleModel.cs @@ -0,0 +1,38 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Core; +// ReSharper disable MemberCanBePrivate.Global + +namespace Microsoft.Python.Analysis.Caching.Models { + //[Serializable] + //internal sealed class SubmoduleModel: MemberModel { + // public string UniqueId { get; set; } + // public string FilePath { get; set; } + // public string Documentation { get; set; } + + // public SubmoduleModel(IPythonModule m, IServiceContainer services) { + // var uniqueId = m.GetUniqueId(services); + // Id = uniqueId.GetStableHash(); + // UniqueId = uniqueId; + // Name = m.Name; + // QualifiedName = m.QualifiedName; + // FilePath = m.FilePath; + // Documentation = m.Documentation; + // } + //} +} diff --git a/src/Caching/Impl/ModuleDatabase.cs b/src/Caching/Impl/ModuleDatabase.cs index 53e6ab04b..75f423834 100644 --- a/src/Caching/Impl/ModuleDatabase.cs +++ b/src/Caching/Impl/ModuleDatabase.cs @@ -26,22 +26,18 @@ using Microsoft.Python.Analysis.Modules; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Core; -using Microsoft.Python.Core.Collections; using Microsoft.Python.Core.IO; using Microsoft.Python.Core.Logging; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.Analysis.Caching { internal sealed class ModuleDatabase : IModuleDatabaseService { - private const int DatabaseFormatVersion = 1; - private readonly Dictionary _dependencies = new Dictionary(); private readonly object _lock = new object(); private readonly IServiceContainer _services; private readonly ILogger _log; private readonly IFileSystem _fs; - private readonly string _databaseFolder; public ModuleDatabase(IServiceContainer services) { _services = services; @@ -49,9 +45,13 @@ public ModuleDatabase(IServiceContainer services) { _fs = services.GetService(); var cfs = services.GetService(); - _databaseFolder = Path.Combine(cfs.CacheFolder, $"analysis.v{DatabaseFormatVersion}"); + CacheFolder = Path.Combine(cfs.CacheFolder, $"{CacheFolderBaseName}{DatabaseFormatVersion}"); } + public string CacheFolderBaseName => "analysis.v"; + public int DatabaseFormatVersion => 2; + public string CacheFolder { get; } + /// /// Retrieves dependencies from the module persistent state. /// @@ -115,10 +115,8 @@ public bool ModuleExistsInStorage(string moduleName, string filePath) { for (var retries = 50; retries > 0; --retries) { try { - lock (_lock) { - var dbPath = FindDatabaseFile(moduleName, filePath); - return !string.IsNullOrEmpty(dbPath); - } + var dbPath = FindDatabaseFile(moduleName, filePath); + return !string.IsNullOrEmpty(dbPath); } catch (Exception ex) when (ex is IOException || ex is UnauthorizedAccessException) { Thread.Sleep(10); } @@ -126,12 +124,6 @@ public bool ModuleExistsInStorage(string moduleName, string filePath) { return false; } - public void Clear() { - lock (_lock) { - _dependencies.Clear(); - } - } - private void StoreModuleAnalysis(IDocumentAnalysis analysis, CancellationToken cancellationToken = default) { var cachingLevel = GetCachingLevel(); if (cachingLevel == AnalysisCachingLevel.None) { @@ -146,26 +138,24 @@ private void StoreModuleAnalysis(IDocumentAnalysis analysis, CancellationToken c Exception ex = null; for (var retries = 50; retries > 0; --retries) { - lock (_lock) { - cancellationToken.ThrowIfCancellationRequested(); - try { - if (!_fs.DirectoryExists(_databaseFolder)) { - _fs.CreateDirectory(_databaseFolder); - } + cancellationToken.ThrowIfCancellationRequested(); + try { + if (!_fs.DirectoryExists(CacheFolder)) { + _fs.CreateDirectory(CacheFolder); + } - cancellationToken.ThrowIfCancellationRequested(); - using (var db = new LiteDatabase(Path.Combine(_databaseFolder, $"{model.UniqueId}.db"))) { - var modules = db.GetCollection("modules"); - modules.Upsert(model); - return; - } - } catch (Exception ex1) when (ex1 is IOException || ex1 is UnauthorizedAccessException) { - ex = ex1; - Thread.Sleep(10); - } catch (Exception ex2) { - ex = ex2; - break; + cancellationToken.ThrowIfCancellationRequested(); + using (var db = new LiteDatabase(Path.Combine(CacheFolder, $"{model.UniqueId}.db"))) { + var modules = db.GetCollection("modules"); + modules.Upsert(model); + return; } + } catch (Exception ex1) when (ex1 is IOException || ex1 is UnauthorizedAccessException) { + ex = ex1; + Thread.Sleep(10); + } catch (Exception ex2) { + ex = ex2; + break; } } @@ -190,7 +180,7 @@ private string FindDatabaseFile(string moduleName, string filePath) { } // Try module name as is. - var dbPath = Path.Combine(_databaseFolder, $"{uniqueId}.db"); + var dbPath = Path.Combine(CacheFolder, $"{uniqueId}.db"); if (_fs.FileExists(dbPath)) { return dbPath; } @@ -199,13 +189,13 @@ private string FindDatabaseFile(string moduleName, string filePath) { // Try with the major.minor Python version. var pythonVersion = interpreter.Configuration.Version; - dbPath = Path.Combine(_databaseFolder, $"{uniqueId}({pythonVersion.Major}.{pythonVersion.Minor}).db"); + dbPath = Path.Combine(CacheFolder, $"{uniqueId}({pythonVersion.Major}.{pythonVersion.Minor}).db"); if (_fs.FileExists(dbPath)) { return dbPath; } // Try with just the major Python version. - dbPath = Path.Combine(_databaseFolder, $"{uniqueId}({pythonVersion.Major}).db"); + dbPath = Path.Combine(CacheFolder, $"{uniqueId}({pythonVersion.Major}).db"); return _fs.FileExists(dbPath) ? dbPath : null; } diff --git a/src/Caching/Impl/ModuleFactory.cs b/src/Caching/Impl/ModuleFactory.cs index c66687872..c9794a714 100644 --- a/src/Caching/Impl/ModuleFactory.cs +++ b/src/Caching/Impl/ModuleFactory.cs @@ -100,7 +100,8 @@ private IMember GetMemberFromThisModule(IReadOnlyList memberNames) { } var nextModel = currentModel.GetModel(memberName); - Debug.Assert(nextModel != null, $"Unable to find member {memberName} in module {Module.Name}"); + Debug.Assert(nextModel != null, + $"Unable to find {string.Join(".", memberNames)} in module {Module.Name}"); if (nextModel == null) { return null; } @@ -168,8 +169,8 @@ private IMember GetMember(IMember root, IEnumerable memberNames) { if (mc is IBuiltinsPythonModule builtins) { // Builtins require special handling since there may be 'hidden' names - // like __NoneType__ which need to be mapped to visible types. - member = GetBuiltinMember(builtins, memberName) ?? builtins.Interpreter.UnknownType; + // which need to be mapped to visible types. + member = GetBuiltinMember(builtins, memberName, typeArgs) ?? builtins.Interpreter.UnknownType; } else { member = mc?.GetMember(memberName); // Work around problem that some stubs have incorrectly named tuples. @@ -193,16 +194,18 @@ private IMember GetMember(IMember root, IEnumerable memberNames) { return member; } - private IMember GetBuiltinMember(IBuiltinsPythonModule builtins, string memberName) { + private IMember GetBuiltinMember(IBuiltinsPythonModule builtins, string memberName, IReadOnlyList typeArgs) { if (memberName.StartsWithOrdinal("__")) { memberName = memberName.Substring(2, memberName.Length - 4); } switch (memberName) { - case "NoneType": - return builtins.Interpreter.GetBuiltinType(BuiltinTypeId.NoneType); + case "None": + return builtins.Interpreter.GetBuiltinType(BuiltinTypeId.None); case "Unknown": return builtins.Interpreter.UnknownType; + case "SuperType": + return new PythonSuperType(typeArgs, builtins); } return builtins.GetMember(memberName); } diff --git a/src/Caching/Impl/TypeNames.cs b/src/Caching/Impl/TypeNames.cs index 2d12f482a..6927e2130 100644 --- a/src/Caching/Impl/TypeNames.cs +++ b/src/Caching/Impl/TypeNames.cs @@ -89,9 +89,9 @@ private static void GetObjectTypeFromPrefix(string qualifiedName, ref QualifiedN private static void GetModuleNameAndMembers(string qualifiedName, ref QualifiedNameParts parts, int prefixOffset) { // Strip the prefix, turning i:module:A.B.C into module:A.B.C var typeName = qualifiedName.Substring(prefixOffset); - var moduleSeparatorIndex = typeName.IndexOf(':'); - if (moduleSeparatorIndex < 0) { + if (moduleSeparatorIndex <= 0) { + typeName = typeName.TrimStart(':'); switch (parts.ObjectType) { case ObjectType.Type: case ObjectType.Instance: diff --git a/src/Caching/Test/ClassesTests.cs b/src/Caching/Test/ClassesTests.cs index b046213fb..00aeaa9a3 100644 --- a/src/Caching/Test/ClassesTests.cs +++ b/src/Caching/Test/ClassesTests.cs @@ -65,6 +65,24 @@ def methodB2(self): Baseline.CompareToFile(BaselineFileName, json); } + [TestMethod, Priority(0)] + public async Task PrivateMembers() { + const string code = @" +class A: + _x = 1 + + def _methodA(self): + return True + + @classmethod + def _methodB(self): + return True +"; + var analysis = await GetAnalysisAsync(code); + var model = ModuleModel.FromAnalysis(analysis, Services, AnalysisCachingLevel.Library); + await CompareBaselineAndRestoreAsync(model, analysis.Document); + } + [TestMethod, Priority(0)] public async Task ForwardDeclarations() { const string code = @" diff --git a/src/Caching/Test/CoreTests.cs b/src/Caching/Test/CoreTests.cs index 4b1adaafa..f1bea1eca 100644 --- a/src/Caching/Test/CoreTests.cs +++ b/src/Caching/Test/CoreTests.cs @@ -110,5 +110,37 @@ def func(a): ... var json = ToJson(model); Baseline.CompareToFile(GetBaselineFileNameWithSuffix(is3x ? "3" : "2"), json); } + + [TestMethod, Priority(0)] + public async Task PositionalOnly() { + const string code = @" +x = 'str' + +class C: + x: int + def __init__(self): + self.y = 1 + + def method(self, x, /, y=True): + return func() + + @property + def prop(self) -> int: + return x + +def func(): + return 2.0 + +c = C() +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); + var model = ModuleModel.FromAnalysis(analysis, Services, AnalysisCachingLevel.Library); + //var json = ToJson(model); + //Baseline.CompareToFile(BaselineFileName, json); + + using (var dbModule = CreateDbModule(model, analysis.Document.FilePath)) { + dbModule.Should().HaveSameMembersAs(analysis.Document); + } + } } } diff --git a/src/Caching/Test/LibraryModulesTests.cs b/src/Caching/Test/LibraryModulesTests.cs index 0d0183579..97fc44751 100644 --- a/src/Caching/Test/LibraryModulesTests.cs +++ b/src/Caching/Test/LibraryModulesTests.cs @@ -30,8 +30,10 @@ public class LibraryModulesTests : AnalysisCachingTestBase { public TestContext TestContext { get; set; } [TestInitialize] - public void TestInitialize() - => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + public void TestInitialize() { + AnalysisTimeout = TimeSpan.FromMinutes(5); + TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + } [TestCleanup] public void Cleanup() => TestEnvironmentImpl.TestCleanup(); @@ -97,6 +99,9 @@ public async Task Builtins() { [TestMethod, Priority(0)] public Task Distutils() => TestModule("distutils"); + [TestMethod, Priority(0)] + public Task EmailGenerator() => TestModule("email.generator"); + [TestMethod, Priority(0)] public Task Email() => TestModule("email"); diff --git a/src/Caching/Test/Microsoft.Python.Analysis.Caching.Tests.csproj b/src/Caching/Test/Microsoft.Python.Analysis.Caching.Tests.csproj index 514328cd5..5957991bd 100644 --- a/src/Caching/Test/Microsoft.Python.Analysis.Caching.Tests.csproj +++ b/src/Caching/Test/Microsoft.Python.Analysis.Caching.Tests.csproj @@ -1,6 +1,6 @@  - netcoreapp3.0 + netcoreapp3.1 Microsoft.Python.Analysis.Caching.Tests Microsoft.Python.Analysis.Caching.Tests @@ -17,22 +17,21 @@ - - - + + + all runtime; build; native; contentfiles; analyzers - + - diff --git a/src/Caching/Test/RestoreTests.cs b/src/Caching/Test/RestoreTests.cs new file mode 100644 index 000000000..b7883c0c4 --- /dev/null +++ b/src/Caching/Test/RestoreTests.cs @@ -0,0 +1,115 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Analysis.Caching.Tests.FluentAssertions; +using Microsoft.Python.Analysis.Documents; +using Microsoft.Python.Analysis.Types; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using TestUtilities; + +namespace Microsoft.Python.Analysis.Caching.Tests { + [TestClass] + public class RestoreTests : AnalysisCachingTestBase { + public TestContext TestContext { get; set; } + + [TestInitialize] + public void TestInitialize() + => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + + [TestCleanup] + public void Cleanup() => TestEnvironmentImpl.TestCleanup(); + + [TestMethod, Priority(0)] + public async Task ReturnType() { + const string code = @" +from module2 import func2 +x = func2() +"; + const string mod2Code = @" +class C2: + def M1C2(self): + return 0 + +def func2() -> C2: ... +"; + await TestData.CreateTestSpecificFileAsync("module2.py", mod2Code); + var analysis = await GetAnalysisAsync(code); + + var f2 = analysis.Should().HaveVariable("func2").Which; + var analysis2 = ((IPythonFunctionType)f2.Value).DeclaringModule.Analysis; + + var dbs = new ModuleDatabase(Services, Path.GetDirectoryName(TestData.GetDefaultModulePath())); + Services.AddService(dbs); + await dbs.StoreModuleAnalysisAsync(analysis2, immediate: true, CancellationToken.None); + + await Services.GetService().ResetAnalyzer(); + var doc = Services.GetService().GetDocument(analysis.Document.Uri); + analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + + var func2 = analysis.Should().HaveFunction("func2").Which; + var c2 = func2.Should().HaveSingleOverload() + .Which.Should().HaveReturnType("C2").Which; + + c2.Should().HaveMember("M1C2") + .Which.Should().HaveSingleOverload() + .Which.Should().HaveReturnType(BuiltinTypeId.Int); + } + + [TestMethod, Priority(0)] + public Task Sys() => TestModule("sys"); + + [TestMethod, Priority(0)] + public Task Os() => TestModule("os"); + + [TestMethod, Priority(0)] + public Task Tokenize() => TestModule("tokenize"); + + [TestMethod, Priority(0)] + public Task Numpy() => TestModule("numpy"); + + private async Task TestModule(string name) { + var analysis = await GetAnalysisAsync($"import {name}"); + var sys = analysis.Should().HaveVariable(name).Which; + + await CreateDatabaseAsync(analysis.Document.Interpreter); + await Services.GetService().ResetAnalyzer(); + var doc = Services.GetService().GetDocument(analysis.Document.Uri); + + var restored = await doc.GetAnalysisAsync(Timeout.Infinite); + var restoredSys = restored.Should().HaveVariable(name).Which; + restoredSys.Value.Should().HaveSameMembersAs(sys.Value); + } + + private async Task CreateDatabaseAsync(IPythonInterpreter interpreter) { + var dbs = new ModuleDatabase(Services, Path.GetDirectoryName(TestData.GetDefaultModulePath())); + Services.AddService(dbs); + + var importedModules = interpreter.ModuleResolution.GetImportedModules(); + foreach (var m in importedModules.Where(m => m.Analysis is LibraryAnalysis)) { + await dbs.StoreModuleAnalysisAsync(m.Analysis, immediate: true); + } + + importedModules = interpreter.TypeshedResolution.GetImportedModules(); + foreach (var m in importedModules.Where(m => m.Analysis is LibraryAnalysis)) { + await dbs.StoreModuleAnalysisAsync(m.Analysis, immediate: true); + } + } + } +} diff --git a/src/Caching/Test/SuperTypeTests.cs b/src/Caching/Test/SuperTypeTests.cs new file mode 100644 index 000000000..2396f01eb --- /dev/null +++ b/src/Caching/Test/SuperTypeTests.cs @@ -0,0 +1,101 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Threading.Tasks; +using Microsoft.Python.Analysis.Caching.Models; +using Microsoft.Python.Analysis.Caching.Tests.FluentAssertions; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using TestUtilities; + +namespace Microsoft.Python.Analysis.Caching.Tests { + [TestClass] + public class SuperTypeTests : AnalysisCachingTestBase { + public TestContext TestContext { get; set; } + + [TestInitialize] + public void TestInitialize() + => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + + [TestCleanup] + public void Cleanup() => TestEnvironmentImpl.TestCleanup(); + + private string BaselineFileName => GetBaselineFileName(TestContext.TestName); + + [TestMethod, Priority(0)] + public async Task ClassesWithSuper() { + const string code = @" +class A: + def methodA(self): + return True + +class B(A): + def methodB(self): + return super() +"; + var analysis = await GetAnalysisAsync(code); + var model = ModuleModel.FromAnalysis(analysis, Services, AnalysisCachingLevel.Library); + + using (var dbModule = CreateDbModule(model, analysis.Document.FilePath)) { + dbModule.Should().HaveSameMembersAs(analysis.Document); + } + } + + [TestMethod, Priority(0)] + public async Task GlobalSuper() { + const string code = @" +class Baze: + def baze_foo(self): + pass + +class Derived(Baze): + def foo(self): + pass + +d = Derived() + +x = super(Derived, d) +"; + var analysis = await GetAnalysisAsync(code); + var model = ModuleModel.FromAnalysis(analysis, Services, AnalysisCachingLevel.Library); + + using (var dbModule = CreateDbModule(model, analysis.Document.FilePath)) { + dbModule.Should().HaveSameMembersAs(analysis.Document); + } + } + + [TestMethod, Priority(0)] + public async Task NamedTupleWithSuper() { + const string code = @" +from typing import NamedTuple + +Point = NamedTuple('Point', ['x', 'y']) + +class ChildPoint(Point): + def foo(self): + pass + +p = ChildPoint() + +x = super(ChildPoint, p) +"; + var analysis = await GetAnalysisAsync(code); + var model = ModuleModel.FromAnalysis(analysis, Services, AnalysisCachingLevel.Library); + + using (var dbModule = CreateDbModule(model, analysis.Document.FilePath)) { + dbModule.Should().HaveSameMembersAs(analysis.Document); + } + } + } +} diff --git a/src/Core/Impl/Collections/ImmutableArray.cs b/src/Core/Impl/Collections/ImmutableArray.cs index 0d8e1f585..b7d274ed2 100644 --- a/src/Core/Impl/Collections/ImmutableArray.cs +++ b/src/Core/Impl/Collections/ImmutableArray.cs @@ -91,6 +91,10 @@ public ImmutableArray Add(T item) { var newItems = _items; var newRef = _ref; + // _ref indicates whether the array "_items" can be re-used when creating new ImmutableArray. + // this is an optimization to reduce array allocation while new elements are kept added at the end of the list + // this is an alternative design compared to Builder model + // (https://docs.microsoft.com/en-us/dotnet/api/system.collections.immutable.immutablearray-1.builder) if (Interlocked.CompareExchange(ref newRef.Count, newCount, _count) != _count || newCount > _items.Length) { var capacity = GetCapacity(newCount); newItems = new T[capacity]; diff --git a/src/Core/Impl/Extensions/ArrayExtensions.cs b/src/Core/Impl/Extensions/ArrayExtensions.cs index 20c6478b2..ce9185487 100644 --- a/src/Core/Impl/Extensions/ArrayExtensions.cs +++ b/src/Core/Impl/Extensions/ArrayExtensions.cs @@ -14,6 +14,7 @@ // permissions and limitations under the License. using System; +using System.Collections.Generic; namespace Microsoft.Python.Core { public static class ArrayExtensions { @@ -36,5 +37,26 @@ public static int IndexOf(this T[] array, TValue value, Func(this TCollection list, TItem item) + where TCollection : ICollection + where TItem : class { + if (item == null) { + return list; + } + + list.Add(item); + return list; + } + + public static TCollection AddIfNotNull(this TCollection list, params TItem[] items) + where TCollection : ICollection + where TItem : class { + foreach (var item in items) { + list.AddIfNotNull(item); + } + + return list; + } } } diff --git a/src/Core/Impl/Extensions/EnumerableExtensions.cs b/src/Core/Impl/Extensions/EnumerableExtensions.cs index aedcd8290..c17900f83 100644 --- a/src/Core/Impl/Extensions/EnumerableExtensions.cs +++ b/src/Core/Impl/Extensions/EnumerableExtensions.cs @@ -166,5 +166,15 @@ public static IEnumerable TraverseDepthFirst(this T root, Func DistinctBy(this IEnumerable source, Func selector) { + var seen = new HashSet(); + + foreach (var item in source) { + if (seen.Add(selector(item))) { + yield return item; + } + } + } } } diff --git a/src/Core/Impl/Extensions/IOExtensions.cs b/src/Core/Impl/Extensions/IOExtensions.cs index 3221295d0..4905a0014 100644 --- a/src/Core/Impl/Extensions/IOExtensions.cs +++ b/src/Core/Impl/Extensions/IOExtensions.cs @@ -14,10 +14,12 @@ // permissions and limitations under the License. using System; +using System.Diagnostics; using System.IO; using System.Linq; using System.Text; using System.Threading; +using Microsoft.Python.Core.Logging; namespace Microsoft.Python.Core.IO { public static class IOExtensions { @@ -79,7 +81,7 @@ public static bool DeleteDirectoryWithRetries(this IFileSystem fs, string path, return !fs.DirectoryExists(path); } - public static FileStream OpenWithRetry(this IFileSystem fs, string file, FileMode mode, FileAccess access, FileShare share) { + public static FileStream OpenWithRetry(this IFileSystem fs, string file, FileMode mode, FileAccess access, FileShare share, ILogger log = null) { // Retry for up to one second var create = mode != FileMode.Open; for (var retries = 100; retries > 0; --retries) { @@ -89,20 +91,23 @@ public static FileStream OpenWithRetry(this IFileSystem fs, string file, FileMod return null; } catch (DirectoryNotFoundException) when (!create) { return null; - } catch (UnauthorizedAccessException) { + } catch (UnauthorizedAccessException uaex) { + log?.Log(TraceEventType.Verbose, "Unable to open file ", file, uaex.Message); Thread.Sleep(10); } catch (IOException) { if (create) { var dir = Path.GetDirectoryName(file); try { fs.CreateDirectory(dir); - } catch (IOException) { - // Cannot create directory for DB, so just bail out - return null; + } catch (IOException ioex) { + log?.Log(TraceEventType.Verbose, "Unable to create directory ", dir, ioex.Message); + Thread.Sleep(10); } + } else { + Thread.Sleep(10); } - Thread.Sleep(10); - } catch (NotSupportedException) { + } catch (NotSupportedException nsx) { + log?.Log(TraceEventType.Verbose, "Unable to open file ", file, nsx.Message); return null; } } @@ -121,21 +126,27 @@ public static string ReadTextWithRetry(this IFileSystem fs, string file) { return null; } - public static void WriteTextWithRetry(this IFileSystem fs, string filePath, string text) { + public static void WriteTextWithRetry(this IFileSystem fs, string filePath, string text, ILogger log = null) { + Exception ex = null; for (var retries = 100; retries > 0; --retries) { try { - using (var stream = fs.OpenWithRetry(filePath, FileMode.Create, FileAccess.Write, FileShare.Read)) { + using (var stream = fs.OpenWithRetry(filePath, FileMode.Create, FileAccess.Write, FileShare.Read, log)) { if (stream != null) { var bytes = Encoding.UTF8.GetBytes(text); stream.Write(bytes, 0, bytes.Length); return; } } - } catch (IOException) { } catch (UnauthorizedAccessException) { - Thread.Sleep(10); + } catch (IOException ioex) { + ex = ioex; + } catch (UnauthorizedAccessException uaex) { + ex = uaex; } + Thread.Sleep(10); } + log?.Log(TraceEventType.Verbose, "Unable to write to ", filePath, ex?.Message ?? "Unknown exception"); + try { fs.DeleteFile(filePath); } catch (IOException) { } catch (UnauthorizedAccessException) { } diff --git a/src/Core/Impl/Extensions/StringExtensions.cs b/src/Core/Impl/Extensions/StringExtensions.cs index be0b1f899..775743c43 100644 --- a/src/Core/Impl/Extensions/StringExtensions.cs +++ b/src/Core/Impl/Extensions/StringExtensions.cs @@ -19,6 +19,8 @@ using System.Globalization; using System.Linq; using System.Runtime.InteropServices; +using System.Security.Cryptography; +using System.Text; using System.Text.RegularExpressions; using Microsoft.Python.Core.Text; @@ -313,5 +315,19 @@ public static int GetStableHash(this string s) { return hash; } } + + /// + /// return string representation of hash of the input string. + /// + /// the string representation of the hash is in the form where it can be used in a file system. + /// + public static string GetHashString(this string input) { + // File name depends on the content so we can distinguish between different versions. + using (var hash = SHA256.Create()) { + return Convert + .ToBase64String(hash.ComputeHash(new UTF8Encoding(encoderShouldEmitUTF8Identifier: false).GetBytes(input))) + .Replace('/', '_').Replace('+', '-'); + } + } } } diff --git a/src/Core/Impl/Extensions/TaskCompletionSourceExtensions.cs b/src/Core/Impl/Extensions/TaskCompletionSourceExtensions.cs index 20f72fbcb..b8ff34a41 100644 --- a/src/Core/Impl/Extensions/TaskCompletionSourceExtensions.cs +++ b/src/Core/Impl/Extensions/TaskCompletionSourceExtensions.cs @@ -18,8 +18,10 @@ namespace Microsoft.Python.Core { public static class TaskCompletionSourceExtensions { + private const int no_delay = -1; + public static CancellationTokenRegistration RegisterForCancellation(this TaskCompletionSource taskCompletionSource, CancellationToken cancellationToken) - => taskCompletionSource.RegisterForCancellation(-1, cancellationToken); + => taskCompletionSource.RegisterForCancellation(millisecondsDelay: no_delay, cancellationToken); public static CancellationTokenRegistration RegisterForCancellation(this TaskCompletionSource taskCompletionSource, int millisecondsDelay, CancellationToken cancellationToken) { if (millisecondsDelay >= 0) { diff --git a/src/Core/Impl/Extensions/TaskExtensions.cs b/src/Core/Impl/Extensions/TaskExtensions.cs index 76643bb4f..e8e3084a5 100644 --- a/src/Core/Impl/Extensions/TaskExtensions.cs +++ b/src/Core/Impl/Extensions/TaskExtensions.cs @@ -21,16 +21,19 @@ namespace Microsoft.Python.Core { public static class TaskExtensions { - public static void SetCompletionResultTo(this Task task, TaskCompletionSource tcs) - => task.ContinueWith(SetCompletionResultToContinuation, tcs, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + public static Task SetCompletionResultTo(this Task task, TaskCompletionSource tcs, bool skipIfCanceled = false) + => task.ContinueWith(t => { + SetCompletionResultToContinuation(t, tcs, skipIfCanceled); + }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); - private static void SetCompletionResultToContinuation(Task task, object state) { - var tcs = (TaskCompletionSource)state; + private static void SetCompletionResultToContinuation(Task task, TaskCompletionSource tcs, bool skipIfCanceled) { switch (task.Status) { case TaskStatus.RanToCompletion: tcs.TrySetResult(task.Result); break; - case TaskStatus.Canceled: + case TaskStatus.Canceled when skipIfCanceled: + break; + case TaskStatus.Canceled when !skipIfCanceled: try { task.GetAwaiter().GetResult(); } catch (OperationCanceledException ex) { @@ -64,7 +67,7 @@ public static void DoNotWait(this Task task) { } var synchronizationContext = SynchronizationContext.Current; - if (synchronizationContext != null && synchronizationContext.GetType() != typeof (SynchronizationContext)) { + if (synchronizationContext != null && synchronizationContext.GetType() != typeof(SynchronizationContext)) { task.ContinueWith(DoNotWaitSynchronizationContextContinuation, synchronizationContext, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); } else { task.ContinueWith(DoNotWaitThreadContinuation, TaskContinuationOptions.ExecuteSynchronously); @@ -87,7 +90,7 @@ private static void DoNotWaitThreadContinuation(Task task) { } private static void DoNotWaitSynchronizationContextContinuation(Task task, object state) { - var context = (SynchronizationContext) state; + var context = (SynchronizationContext)state; context.Post(ReThrowTaskException, task); } @@ -105,7 +108,16 @@ private static void DoNotWaitSynchronizationContextContinuation(Task task, objec /// public static T WaitAndUnwrapExceptions(this Task task) => task.GetAwaiter().GetResult(); - public static Task WaitAsync(this Task task, CancellationToken cancellationToken) - => task.ContinueWith(t => t.GetAwaiter().GetResult(), cancellationToken, TaskContinuationOptions.None, TaskScheduler.Default); + /// + /// Attach new to the given task. + /// + /// this allows caller to have its own cancellation without aborting underlying work. + /// + /// if uses different cancellation token than one given + /// it will throw instead of and + /// Task will be set to faulted rather than cancelled. + /// + public static Task WaitAsync(this Task task, CancellationToken cancellationToken) + => task.ContinueWith(t => t.WaitAndUnwrapExceptions(), cancellationToken, TaskContinuationOptions.None, TaskScheduler.Default); } } diff --git a/src/Core/Impl/IO/PathUtils.cs b/src/Core/Impl/IO/PathUtils.cs index ff73cd966..75ecb202d 100644 --- a/src/Core/Impl/IO/PathUtils.cs +++ b/src/Core/Impl/IO/PathUtils.cs @@ -521,5 +521,30 @@ public static string NormalizePath(string path) { } public static string NormalizePathAndTrim(string path) => TrimEndSeparator(NormalizePath(path)); + + public static string LookPath(IFileSystem fs, string exeName) { + var path = Environment.GetEnvironmentVariable("PATH"); + if (string.IsNullOrWhiteSpace(path)) { + return null; + } + + foreach (var p in path.Split(Path.PathSeparator)) { + var x = Path.Combine(p, exeName); + + if (IsWindows) { + x += ".exe"; // TODO: other extensions? + } + + if (!fs.FileExists(x)) { + continue; + } + + // TODO: check executable on non-Windows platforms. + + return x; + } + + return null; + } } } diff --git a/src/Core/Impl/IO/WithRetries.cs b/src/Core/Impl/IO/WithRetries.cs new file mode 100644 index 000000000..41c7a7a44 --- /dev/null +++ b/src/Core/Impl/IO/WithRetries.cs @@ -0,0 +1,46 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Diagnostics; +using System.IO; +using System.Threading; +using Microsoft.Python.Core.Logging; + +namespace Microsoft.Python.Core.IO { + public static class WithRetries { + public static T Execute(Func a, string errorMessage, ILogger log) { + Exception ex = null; + for (var retries = 50; retries > 0; --retries) { + try { + return a(); + } catch (Exception ex1) when (ex1 is IOException || ex1 is UnauthorizedAccessException) { + ex = ex1; + Thread.Sleep(10); + } catch (Exception ex2) { + ex = ex2; + break; + } + } + if (ex != null) { + log?.Log(TraceEventType.Warning, $"{errorMessage} Exception: {ex.Message}"); + if (ex.IsCriticalException()) { + throw ex; + } + } + return default; + } + } +} diff --git a/src/Core/Impl/Microsoft.Python.Core.csproj b/src/Core/Impl/Microsoft.Python.Core.csproj index f41668645..2fa1f0cd4 100644 --- a/src/Core/Impl/Microsoft.Python.Core.csproj +++ b/src/Core/Impl/Microsoft.Python.Core.csproj @@ -9,7 +9,7 @@ 1701, 1702 - "You may need to supply assembly policy" --> 1701;1702;$(NoWarn) - 7.3 + true @@ -23,7 +23,7 @@ all runtime; build; native; contentfiles; analyzers - + diff --git a/src/Core/Impl/Services/IdleTimeService.cs b/src/Core/Impl/Services/IdleTimeService.cs index fa7d1431f..38021ae28 100644 --- a/src/Core/Impl/Services/IdleTimeService.cs +++ b/src/Core/Impl/Services/IdleTimeService.cs @@ -19,11 +19,15 @@ namespace Microsoft.Python.Core.Services { public sealed class IdleTimeService : IIdleTimeService, IIdleTimeTracker, IDisposable { + private static readonly TimeSpan InitialDelay = TimeSpan.FromMilliseconds(50); + private static readonly TimeSpan Interval = TimeSpan.FromMilliseconds(50); + private static readonly TimeSpan IdleInterval = TimeSpan.FromMilliseconds(100); + private Timer _timer; private DateTime _lastActivityTime; public IdleTimeService() { - _timer = new Timer(OnTimer, this, 50, 50); + _timer = new Timer(OnTimer, this, InitialDelay, Interval); NotifyUserActivity(); } @@ -32,11 +36,12 @@ public IdleTimeService() { public void Dispose() { _timer?.Dispose(); _timer = null; + Closing?.Invoke(this, EventArgs.Empty); } private void OnTimer(object state) { - if ((DateTime.Now - _lastActivityTime).TotalMilliseconds >= 100 && _timer != null) { + if (_timer != null && (DateTime.Now - _lastActivityTime) >= IdleInterval) { Idle?.Invoke(this, EventArgs.Empty); } } diff --git a/src/Core/Impl/Text/Position.cs b/src/Core/Impl/Text/Position.cs index d7ad18649..5f0a4de11 100644 --- a/src/Core/Impl/Text/Position.cs +++ b/src/Core/Impl/Text/Position.cs @@ -18,7 +18,7 @@ namespace Microsoft.Python.Core.Text { [Serializable] - public struct Position { + public struct Position : IEquatable { /// /// Line position in a document (zero-based). /// @@ -39,7 +39,14 @@ public struct Position { public static bool operator >(Position p1, Position p2) => p1.line > p2.line || p1.line == p2.line && p1.character > p2.character; public static bool operator <(Position p1, Position p2) => p1.line < p2.line || p1.line == p2.line && p1.character < p2.character; + public static bool operator ==(Position p1, Position p2) => p1.Equals(p2); + public static bool operator !=(Position p1, Position p2) => !p1.Equals(p2); + public bool Equals(Position other) => line == other.line && character == other.character; + + public override bool Equals(object obj) => obj is Position other ? Equals(other) : false; + + public override int GetHashCode() => 0; public override string ToString() => $"({line}, {character})"; } } diff --git a/src/Core/Impl/Threading/AsyncAutoResetEvent.cs b/src/Core/Impl/Threading/AsyncAutoResetEvent.cs index 035b30920..b33ce47f9 100644 --- a/src/Core/Impl/Threading/AsyncAutoResetEvent.cs +++ b/src/Core/Impl/Threading/AsyncAutoResetEvent.cs @@ -57,7 +57,7 @@ public void Set() { } } - if (!_isSignaled && (waiterToRelease == default || waiterToRelease.Task.IsCompleted)) { + if (!_isSignaled && (waiterToRelease == null || waiterToRelease.Task.IsCompleted)) { _isSignaled = true; } } diff --git a/src/Core/Impl/Threading/PriorityProducerConsumer.cs b/src/Core/Impl/Threading/PriorityProducerConsumer.cs index 21cfa8afa..3bebd4d76 100644 --- a/src/Core/Impl/Threading/PriorityProducerConsumer.cs +++ b/src/Core/Impl/Threading/PriorityProducerConsumer.cs @@ -25,7 +25,7 @@ namespace Microsoft.Python.Core.Threading { public sealed class PriorityProducerConsumer : IDisposable { private readonly int _maxPriority; private readonly object _syncObj; - private readonly List[] _queues; + private readonly LinkedList[] _queues; private readonly Queue _pendingTasks; private readonly DisposeToken _disposeToken; private readonly bool _excludeDuplicates; @@ -45,9 +45,9 @@ public int Count { public PriorityProducerConsumer(int maxPriority = 1, bool excludeDuplicates = false, IEqualityComparer comparer = null) { _maxPriority = maxPriority; - _queues = new List[maxPriority]; + _queues = new LinkedList[maxPriority]; for (var i = 0; i < _queues.Length; i++) { - _queues[i] = new List(); + _queues[i] = new LinkedList(); } _syncObj = new object(); @@ -87,7 +87,7 @@ public void Produce(T value, int priority = 0) { RemoveExistingValue(value, ref priority); } - _queues[priority].Add(value); + _queues[priority].AddLast(value); _firstAvailablePriority = Math.Min(_firstAvailablePriority, priority); } } @@ -99,14 +99,17 @@ private void RemoveExistingValue(T value, ref int priority) { lock (_syncObj) { for (var i = 0; i < _maxPriority; i++) { var queue = _queues[i]; - for (var j = 0; j < queue.Count; j++) { + var current = queue.First; + while (current != null) { // Check if value is scheduled already - // There can be no more than one - if (_comparer.Equals(queue[j], value)) { + // There can be no more than one duplicate + if (_comparer.Equals(current.Value, value)) { priority = Math.Min(i, priority); - queue.RemoveAt(j); + queue.Remove(current); return; } + + current = current.Next; } } } @@ -127,15 +130,16 @@ public Task ConsumeAsync(CancellationToken cancellationToken = default) { Debug.Assert(_pendingTasks.Count == 0 || _firstAvailablePriority == _maxPriority); if (_firstAvailablePriority < _maxPriority) { var queue = _queues[_firstAvailablePriority]; - var result = queue[0]; - queue.RemoveAt(0); + var result = queue.First; + queue.RemoveFirst(); + if (queue.Count == 0) { do { _firstAvailablePriority++; } while (_firstAvailablePriority < _maxPriority && _queues[_firstAvailablePriority].Count == 0); } - return Task.FromResult(result); + return Task.FromResult(result.Value); } pendingTcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); @@ -156,7 +160,7 @@ private static void RegisterCancellation(CancellationToken cancellationToken, Pe .UnregisterOnCompletion(pendingTcs.Task); private static void CancelCallback(object state) { - var cancelState = (CancelState) state; + var cancelState = (CancelState)state; cancelState.Pending.Release()?.TrySetCanceled(cancelState.CancellationToken); } diff --git a/src/Core/Impl/Threading/SingleThreadSynchronizationContext.cs b/src/Core/Impl/Threading/SingleThreadSynchronizationContext.cs index 9f0afe0bb..5c5ef4829 100644 --- a/src/Core/Impl/Threading/SingleThreadSynchronizationContext.cs +++ b/src/Core/Impl/Threading/SingleThreadSynchronizationContext.cs @@ -20,7 +20,7 @@ namespace Microsoft.Python.Core.Threading { public class SingleThreadSynchronizationContext : SynchronizationContext, IDisposable { - private readonly ConcurrentQueue> _queue = new ConcurrentQueue>(); + private readonly ConcurrentQueue<(SendOrPostCallback callback, object state)> _queue = new ConcurrentQueue<(SendOrPostCallback, object)>(); private readonly ManualResetEventSlim _workAvailable = new ManualResetEventSlim(false); private readonly CancellationTokenSource _cts = new CancellationTokenSource(); @@ -29,7 +29,7 @@ public SingleThreadSynchronizationContext() { } public override void Post(SendOrPostCallback d, object state) { - _queue.Enqueue(new Tuple(d, state)); + _queue.Enqueue((d, state)); _workAvailable.Set(); } @@ -41,9 +41,10 @@ private void QueueWorker() { if (_cts.IsCancellationRequested) { break; } - while (_queue.TryDequeue(out var t)) { - t.Item1(t.Item2); + while (_queue.TryDequeue(out var entry)) { + entry.callback(entry.state); } + _workAvailable.Reset(); } } diff --git a/src/Core/Impl/Threading/TaskQueue.cs b/src/Core/Impl/Threading/TaskQueue.cs new file mode 100644 index 000000000..ae6995430 --- /dev/null +++ b/src/Core/Impl/Threading/TaskQueue.cs @@ -0,0 +1,77 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Threading; + +namespace Microsoft.Python.Core.Threading { + // https://stackoverflow.com/questions/1656404/c-sharp-producer-consumer + // http://www.albahari.com/threading/part4.aspx#_Wait_and_Pulse + public sealed class TaskQueue : IDisposable { + private readonly object _lock = new object(); + private readonly Thread[] _workers; + private readonly Queue _queue = new Queue(); + + public TaskQueue(int maxWorkers) { + _workers = new Thread[maxWorkers]; + // Create and start a separate thread for each worker + for (var i = 0; i < _workers.Length; i++) { + (_workers[i] = new Thread(Consume)).Start(); + } + } + + public void Dispose() { + // Enqueue one null task per worker to make each exit. + foreach (var worker in _workers) { + Enqueue(null); + } + foreach (var worker in _workers) { + worker.Join(); + } + } + + public void Enqueue(Action action, bool immediate = true) { + lock (_lock) { + _queue.Enqueue(action); + if (immediate) { + Monitor.PulseAll(_lock); + } + } + } + + public void ProcessQueue() { + lock (_lock) { + Monitor.PulseAll(_lock); + } + } + + private void Consume() { + while (true) { + Action action; + lock (_lock) { + while (_queue.Count == 0) { + Monitor.Wait(_lock); + } + action = _queue.Dequeue(); + } + if (action == null) { + break; + } + action(); + } + } + } +} diff --git a/src/Core/Test/Microsoft.Python.Core.Tests.csproj b/src/Core/Test/Microsoft.Python.Core.Tests.csproj index f475b7992..7104be848 100644 --- a/src/Core/Test/Microsoft.Python.Core.Tests.csproj +++ b/src/Core/Test/Microsoft.Python.Core.Tests.csproj @@ -1,6 +1,6 @@  - netcoreapp3.0 + netcoreapp3.1 Microsoft.Python.Core.Tests Microsoft.Python.Core.Tests @@ -27,10 +27,10 @@ - - - - + + + + all runtime; build; native; contentfiles; analyzers diff --git a/src/Core/Test/PriorityProducerConsumerTest.cs b/src/Core/Test/PriorityProducerConsumerTest.cs index a0a5132ac..72ec44de5 100644 --- a/src/Core/Test/PriorityProducerConsumerTest.cs +++ b/src/Core/Test/PriorityProducerConsumerTest.cs @@ -14,6 +14,8 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System; +using System.Threading; using System.Threading.Tasks; using FluentAssertions; using Microsoft.Python.Core.Threading; @@ -24,102 +26,96 @@ namespace Microsoft.Python.Core.Tests { public class PriorityProducerConsumerTest { [TestMethod, Priority(0)] public void PriorityProducerConsumer_NoPending() { - using (var ppc = new PriorityProducerConsumer()) { - ppc.Produce(5); - var consumerTask = ppc.ConsumeAsync(); - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); - Assert.AreEqual(5, consumerTask.Result); - } + using var ppc = new PriorityProducerConsumer(); + + ppc.Produce(5); + var consumerTask = ppc.ConsumeAsync(); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); + Assert.AreEqual(5, consumerTask.Result); } [TestMethod, Priority(0)] public void PriorityProducerConsumer_NoPending_Priority1() { - using (var ppc = new PriorityProducerConsumer(2)) { - ppc.Produce(5); - ppc.Produce(6, 1); - var consumerTask = ppc.ConsumeAsync(); - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); - Assert.AreEqual(5, consumerTask.Result); - } + using var ppc = new PriorityProducerConsumer(2); + ppc.Produce(5); + ppc.Produce(6, 1); + var consumerTask = ppc.ConsumeAsync(); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); + Assert.AreEqual(5, consumerTask.Result); } [TestMethod, Priority(0)] public void PriorityProducerConsumer_NoPending_Priority2() { - using (var ppc = new PriorityProducerConsumer(2)) { - ppc.Produce(6, 1); - ppc.Produce(5); - var consumerTask = ppc.ConsumeAsync(); - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); - Assert.AreEqual(5, consumerTask.Result); - } + using var ppc = new PriorityProducerConsumer(2); + ppc.Produce(6, 1); + ppc.Produce(5); + var consumerTask = ppc.ConsumeAsync(); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); + Assert.AreEqual(5, consumerTask.Result); } [TestMethod, Priority(0)] public void PriorityProducerConsumer_NoPending_Duplicates1() { - using (var ppc = new PriorityProducerConsumer(3, true)) { - ppc.Produce(5, 2); - ppc.Produce(6, 1); - ppc.Produce(5); - var consumerTask1 = ppc.ConsumeAsync(); - var consumerTask2 = ppc.ConsumeAsync(); - var consumerTask3 = ppc.ConsumeAsync(); - - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask3.Status); - Assert.AreEqual(5, consumerTask1.Result); - Assert.AreEqual(6, consumerTask2.Result); - } + using var ppc = new PriorityProducerConsumer(3, true); + ppc.Produce(5, 2); + ppc.Produce(6, 1); + ppc.Produce(5); + var consumerTask1 = ppc.ConsumeAsync(); + var consumerTask2 = ppc.ConsumeAsync(); + var consumerTask3 = ppc.ConsumeAsync(); + + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask3.Status); + Assert.AreEqual(5, consumerTask1.Result); + Assert.AreEqual(6, consumerTask2.Result); } [TestMethod, Priority(0)] public void PriorityProducerConsumer_NoPending_Duplicates2() { - using (var ppc = new PriorityProducerConsumer(3, true)) { - ppc.Produce(5); - ppc.Produce(6, 1); - ppc.Produce(5, 2); - var consumerTask1 = ppc.ConsumeAsync(); - var consumerTask2 = ppc.ConsumeAsync(); - var consumerTask3 = ppc.ConsumeAsync(); - - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask3.Status); - Assert.AreEqual(5, consumerTask1.Result); - Assert.AreEqual(6, consumerTask2.Result); - } + using var ppc = new PriorityProducerConsumer(3, true); + ppc.Produce(5); + ppc.Produce(6, 1); + ppc.Produce(5, 2); + var consumerTask1 = ppc.ConsumeAsync(); + var consumerTask2 = ppc.ConsumeAsync(); + var consumerTask3 = ppc.ConsumeAsync(); + + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask3.Status); + Assert.AreEqual(5, consumerTask1.Result); + Assert.AreEqual(6, consumerTask2.Result); } [TestMethod, Priority(0)] public void PriorityProducerConsumer_NoPending_Duplicates3() { - using (var ppc = new PriorityProducerConsumer(3, true)) { - ppc.Produce(5, 1); - ppc.Produce(6, 1); - ppc.Produce(5, 1); - var consumerTask1 = ppc.ConsumeAsync(); - var consumerTask2 = ppc.ConsumeAsync(); - var consumerTask3 = ppc.ConsumeAsync(); - - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask3.Status); - Assert.AreEqual(6, consumerTask1.Result); - Assert.AreEqual(5, consumerTask2.Result); - } + using var ppc = new PriorityProducerConsumer(3, true); + ppc.Produce(5, 1); + ppc.Produce(6, 1); + ppc.Produce(5, 1); + var consumerTask1 = ppc.ConsumeAsync(); + var consumerTask2 = ppc.ConsumeAsync(); + var consumerTask3 = ppc.ConsumeAsync(); + + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask3.Status); + Assert.AreEqual(6, consumerTask1.Result); + Assert.AreEqual(5, consumerTask2.Result); } [TestMethod, Priority(0)] public async Task PriorityProducerConsumer_Pending() { - using (var ppc = new PriorityProducerConsumer()) { - var consumerTask = ppc.ConsumeAsync(); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask.Status); + using var ppc = new PriorityProducerConsumer(); + var consumerTask = ppc.ConsumeAsync(); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask.Status); - ppc.Produce(5); - await consumerTask; + ppc.Produce(5); + await consumerTask; - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); - Assert.AreEqual(5, consumerTask.Result); - } + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); + Assert.AreEqual(5, consumerTask.Result); } [TestMethod, Priority(0)] @@ -138,72 +134,116 @@ public async Task PriorityProducerConsumer_Pending_Dispose() { [TestMethod, Priority(0)] public async Task PriorityProducerConsumer_Pending_Priority1() { - using (var ppc = new PriorityProducerConsumer(2)) { - var consumerTask = ppc.ConsumeAsync(); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask.Status); + using var ppc = new PriorityProducerConsumer(2); + var consumerTask = ppc.ConsumeAsync(); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask.Status); - ppc.Produce(5); - ppc.Produce(6, 1); - await consumerTask; + ppc.Produce(5); + ppc.Produce(6, 1); + await consumerTask; - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); - Assert.AreEqual(5, consumerTask.Result); - } + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); + Assert.AreEqual(5, consumerTask.Result); } [TestMethod, Priority(0)] public async Task PriorityProducerConsumer_Pending_Priority2() { - using (var ppc = new PriorityProducerConsumer(2)) { - var consumerTask1 = ppc.ConsumeAsync(); - var consumerTask2 = ppc.ConsumeAsync(); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask1.Status); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask2.Status); + using var ppc = new PriorityProducerConsumer(2); + var consumerTask1 = ppc.ConsumeAsync(); + var consumerTask2 = ppc.ConsumeAsync(); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask1.Status); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask2.Status); - ppc.Produce(6, 1); - await consumerTask1; + ppc.Produce(6, 1); + await consumerTask1; - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask2.Status); - Assert.AreEqual(6, consumerTask1.Result); + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask1.Status); + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask2.Status); + Assert.AreEqual(6, consumerTask1.Result); - ppc.Produce(5); - await consumerTask2; + ppc.Produce(5); + await consumerTask2; - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); - Assert.AreEqual(5, consumerTask2.Result); - } + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask2.Status); + Assert.AreEqual(5, consumerTask2.Result); } [TestMethod, Priority(0)] public async Task PriorityProducerConsumer_Pending_Priority3() { - using (var ppc = new PriorityProducerConsumer(2)) { - var values = new int[3]; - var tcsConsumer = new TaskCompletionSource(); - var tcsProducer = new TaskCompletionSource(); - var consumerTask = Task.Run(async () => { - for (var i = 0; i < 3; i++) { - var task = ppc.ConsumeAsync(); - tcsConsumer.TrySetResult(true); - values[i] = await task; - await tcsProducer.Task; - } - }); - - Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask.Status); - - await tcsConsumer.Task; - ppc.Produce(5, 1); - ppc.Produce(6, 1); - ppc.Produce(7); - tcsProducer.SetResult(false); - - await consumerTask; - - Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); - Assert.AreEqual(5, values[0]); - Assert.AreEqual(7, values[1]); - Assert.AreEqual(6, values[2]); - } + using var ppc = new PriorityProducerConsumer(2); + var values = new int[3]; + var tcsConsumer = new TaskCompletionSource(); + var tcsProducer = new TaskCompletionSource(); + var consumerTask = Task.Run(async () => { + for (var i = 0; i < 3; i++) { + var task = ppc.ConsumeAsync(); + tcsConsumer.TrySetResult(true); + values[i] = await task; + await tcsProducer.Task; + } + }); + + Assert.AreEqual(TaskStatus.WaitingForActivation, consumerTask.Status); + + await tcsConsumer.Task; + ppc.Produce(5, 1); + ppc.Produce(6, 1); + ppc.Produce(7); + tcsProducer.SetResult(false); + + await consumerTask; + + Assert.AreEqual(TaskStatus.RanToCompletion, consumerTask.Status); + Assert.AreEqual(5, values[0]); + Assert.AreEqual(7, values[1]); + Assert.AreEqual(6, values[2]); + } + + [TestMethod, Priority(0)] + public async Task PriorityProducerConsumer_ExcludeDuplicates() { + using var ppc = new PriorityProducerConsumer(maxPriority: 2, excludeDuplicates: true); + + ppc.Produce(value: 1, priority: 0); + ppc.Produce(value: 2, priority: 0); + ppc.Produce(value: 3, priority: 0); + ppc.Produce(value: 1, priority: 1); + ppc.Produce(value: 2, priority: 1); + + var data1 = await ppc.ConsumeAsync(CancellationToken.None); + data1.Should().Be(3); + + var data2 = await ppc.ConsumeAsync(CancellationToken.None); + data2.Should().Be(1); + + var data3 = await ppc.ConsumeAsync(CancellationToken.None); + data3.Should().Be(2); + } + + [TestMethod, Priority(0)] + public async Task PriorityProducerConsumer_ExcludeDuplicates_HigherPriority() { + using var ppc = new PriorityProducerConsumer(maxPriority: 2, excludeDuplicates: true); + + ppc.Produce(value: 1, priority: 1); + ppc.Produce(value: 2, priority: 1); + ppc.Produce(value: 3, priority: 1); + ppc.Produce(value: 1, priority: 0); + ppc.Produce(value: 2, priority: 0); + + var data1 = await ppc.ConsumeAsync(CancellationToken.None); + data1.Should().Be(1); + + var data2 = await ppc.ConsumeAsync(CancellationToken.None); + data2.Should().Be(2); + + var data3 = await ppc.ConsumeAsync(CancellationToken.None); + data3.Should().Be(3); + } + + [TestMethod, Priority(0)] + public void PriorityProducerConsumer_MaxPriority() { + using var ppc = new PriorityProducerConsumer(maxPriority: 2, excludeDuplicates: false); + + Assert.ThrowsException(() => ppc.Produce(value: 1, priority: 2)); } } } diff --git a/src/LanguageServer/Impl/CodeActions/CodeActionSettings.cs b/src/LanguageServer/Impl/CodeActions/CodeActionSettings.cs new file mode 100644 index 000000000..ce536f1e6 --- /dev/null +++ b/src/LanguageServer/Impl/CodeActions/CodeActionSettings.cs @@ -0,0 +1,51 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; + +namespace Microsoft.Python.LanguageServer.CodeActions { + public sealed class CodeActionSettings { + public static readonly CodeActionSettings Default = new CodeActionSettings(refactoring: null, quickFix: null); + + private readonly IReadOnlyDictionary _refactoring; + private readonly IReadOnlyDictionary _quickFix; + + public CodeActionSettings(IReadOnlyDictionary refactoring, IReadOnlyDictionary quickFix) { + _refactoring = refactoring ?? new Dictionary(); + _quickFix = quickFix ?? new Dictionary(); + } + + public T GetRefactoringOption(string key, T defaultValue) => GetOption(_refactoring, key, defaultValue); + public T GetQuickFixOption(string key, T defaultValue) => GetOption(_quickFix, key, defaultValue); + + private T GetOption(IReadOnlyDictionary map, string key, T defaultValue) { + if (key == null) { + // invalid key + return defaultValue; + } + + if (map.TryGetValue(key, out var value)) { + if (value == null) { + // if value is explicitly set to null, then return null + return (T)value; + } + + return (value is T typedValue) ? typedValue : defaultValue; + } + + return defaultValue; + } + } +} diff --git a/src/LanguageServer/Impl/CodeActions/MissingImportCodeActionProvider.cs b/src/LanguageServer/Impl/CodeActions/MissingImportCodeActionProvider.cs new file mode 100644 index 000000000..d48804375 --- /dev/null +++ b/src/LanguageServer/Impl/CodeActions/MissingImportCodeActionProvider.cs @@ -0,0 +1,735 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Data; +using System.Diagnostics; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis; +using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Analysis.Analyzer.Expressions; +using Microsoft.Python.Analysis.Core.DependencyResolution; +using Microsoft.Python.Analysis.Core.Interpreter; +using Microsoft.Python.Analysis.Diagnostics; +using Microsoft.Python.Analysis.Modules; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core; +using Microsoft.Python.Core.Collections; +using Microsoft.Python.Core.Text; +using Microsoft.Python.LanguageServer.Diagnostics; +using Microsoft.Python.LanguageServer.Indexing; +using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.LanguageServer.Utilities; +using Microsoft.Python.Parsing.Ast; +using Range = Microsoft.Python.Core.Text.Range; + +namespace Microsoft.Python.LanguageServer.CodeActions { + internal sealed class MissingImportCodeActionProvider : IQuickFixCodeActionProvider { + public static readonly IQuickFixCodeActionProvider Instance = new MissingImportCodeActionProvider(); + + // right now, it is a static. in future, we might consider giving an option to users to customize this list + // also, right now, it is text based. so if module has same name, they will get same suggestion even if + // the module is not something user expected + private static readonly Dictionary WellKnownAbbreviationMap = new Dictionary() { + { "numpy", "np" }, + { "pandas", "pd" }, + { "tensorflow", "tf" }, + { "matplotlib.pyplot", "plt" }, + { "matplotlib", "mpl" }, + { "math", "m" }, + { "scipy.io", "spio" }, + { "scipy", "sp" }, + }; + + private MissingImportCodeActionProvider() { + } + + public ImmutableArray FixableDiagnostics => ImmutableArray.Create( + ErrorCodes.UndefinedVariable, ErrorCodes.VariableNotDefinedGlobally, ErrorCodes.VariableNotDefinedNonLocal); + + public async Task> GetCodeActionsAsync(IDocumentAnalysis analysis, CodeActionSettings settings, DiagnosticsEntry diagnostic, CancellationToken cancellationToken) { + if (!settings.GetQuickFixOption("addimports", true)) { + return Enumerable.Empty(); + } + + var finder = new ExpressionFinder(analysis.Ast, new FindExpressionOptions() { Names = true }); + var node = finder.GetExpression(diagnostic.SourceSpan); + if (!(node is NameExpression nex)) { + return Enumerable.Empty(); + } + + var identifier = nex.Name; + if (string.IsNullOrEmpty(identifier)) { + return Enumerable.Empty(); + } + + var codeActions = new List(); + var diagnostics = new[] { diagnostic.ToDiagnostic() }; + + // see whether it is one of abbreviation we specialize + foreach (var moduleFullName in WellKnownAbbreviationMap.Where(kv => kv.Value == identifier).Select(kv => kv.Key)) { + var moduleName = GetModuleName(moduleFullName); + + await GetCodeActionsAsync(analysis, diagnostics, new Input(node, moduleName, moduleFullName), codeActions, cancellationToken); + } + + // add then search given name as it is + await GetCodeActionsAsync(analysis, diagnostics, new Input(node, identifier), codeActions, cancellationToken); + + return codeActions; + + string GetModuleName(string moduleFullName) { + var index = moduleFullName.LastIndexOf("."); + return index < 0 ? moduleFullName : moduleFullName.Substring(index + 1); + } + } + + private async Task GetCodeActionsAsync(IDocumentAnalysis analysis, + Diagnostic[] diagnostics, + Input input, + List codeActions, + CancellationToken cancellationToken) { + var importFullNameMap = new Dictionary(); + await AddCandidatesFromIndexAsync(analysis, input.Identifier, importFullNameMap, cancellationToken); + + var interpreter = analysis.Document.Interpreter; + var pathResolver = interpreter.ModuleResolution.CurrentPathResolver; + + // find installed modules matching the given name. this will include submodules + var languageVersion = Parsing.PythonLanguageVersionExtensions.ToVersion(interpreter.LanguageVersion); + var includeImplicit = !ModulePath.PythonVersionRequiresInitPyFiles(languageVersion); + + foreach (var moduleFullName in pathResolver.GetAllImportableModulesByName(input.Identifier, includeImplicit)) { + cancellationToken.ThrowIfCancellationRequested(); + importFullNameMap[moduleFullName] = new ImportInfo(moduleImported: false, memberImported: false, isModule: true); + } + + // find members matching the given name from modules already loaded. + var moduleInfo = new ModuleInfo(analysis); + foreach (var module in interpreter.ModuleResolution.GetImportedModules(cancellationToken)) { + if (module.ModuleType == ModuleType.Unresolved) { + continue; + } + + // module name is full module name that you can use in import xxxx directly + CollectCandidates(moduleInfo.Reset(module), input.Identifier, importFullNameMap, cancellationToken); + Debug.Assert(moduleInfo.NameParts.Count == 1 && moduleInfo.NameParts[0] == module.Name); + } + + // check quick bail out case where we know what module we are looking for + if (input.ModuleFullNameOpt != null) { + if (importFullNameMap.ContainsKey(input.ModuleFullNameOpt)) { + // add code action if the module exist, otherwise, bail out empty + codeActions.AddIfNotNull(CreateCodeAction(analysis, input.Context, input.ModuleFullNameOpt, diagnostics, locallyInserted: false, cancellationToken)); + } + return; + } + + // regular case + FilterCandidatesBasedOnContext(analysis, input.Context, importFullNameMap, cancellationToken); + + // this will create actual code fix with certain orders + foreach (var fullName in OrderFullNames(importFullNameMap)) { + cancellationToken.ThrowIfCancellationRequested(); + codeActions.AddIfNotNull(CreateCodeAction(analysis, input.Context, fullName, diagnostics, locallyInserted: false, cancellationToken)); + } + } + + private void FilterCandidatesBasedOnContext(IDocumentAnalysis analysis, Node node, Dictionary importFullNameMap, CancellationToken cancellationToken) { + var ancestors = GetAncestorsOrThis(analysis.Ast.Body, node, cancellationToken); + var index = ancestors.LastIndexOf(node); + if (index <= 0) { + // nothing to filter on + return; + } + + var parent = ancestors[index - 1]; + if (!(parent is CallExpression)) { + // nothing to filter on + return; + } + + // do simple filtering + // remove all modules from candidates + foreach (var kv in importFullNameMap.ToList()) { + if (kv.Value.IsModule) { + importFullNameMap.Remove(kv.Key); + } + } + } + + private IEnumerable OrderFullNames(Dictionary importFullNameMap) { + // use some heuristic to improve code fix ordering + + // put simple name module at the top + foreach (var fullName in OrderImportNames(importFullNameMap.Where(FilterSimpleName).Select(kv => kv.Key))) { + importFullNameMap.Remove(fullName); + yield return fullName; + } + + // heuristic is we put entries with decl without any exports (imported member with __all__) at the top + // such as array. another example will be chararray. + // this will make numpy chararray at the top and numpy defchararray at the bottom. + // if we want, we can add more info to hide intermediate ones. + // for example, numpy.chararry is __all__.extended from numpy.core.chararray and etc. + // so we could leave only numpy.chararray and remove ones like numpy.core.chararray and etc. but for now, + // we show all those but in certain order so that numpy.chararray shows up top + // this heuristic still has issue with something like os.path.join since no one import macpath, macpath join shows up high + var sourceDeclarationFullNames = importFullNameMap.Where(kv => kv.Value.Symbol != null) + .GroupBy(kv => kv.Value.Symbol.Definition, LocationInfo.FullComparer) + .Where(FilterSourceDeclarations) + .Select(g => g.First().Key); + + foreach (var fullName in OrderImportNames(sourceDeclarationFullNames)) { + importFullNameMap.Remove(fullName); + yield return fullName; + } + + // put modules that are imported next + foreach (var fullName in OrderImportNames(importFullNameMap.Where(FilterImportedModules).Select(kv => kv.Key))) { + importFullNameMap.Remove(fullName); + yield return fullName; + } + + // put members that are imported next + foreach (var fullName in OrderImportNames(importFullNameMap.Where(FilterImportedMembers).Select(kv => kv.Key))) { + importFullNameMap.Remove(fullName); + yield return fullName; + } + + // put members whose module is imported next + foreach (var fullName in OrderImportNames(importFullNameMap.Where(FilterImportedModuleMembers).Select(kv => kv.Key))) { + importFullNameMap.Remove(fullName); + yield return fullName; + } + + // put things left here. + foreach (var fullName in OrderImportNames(importFullNameMap.Select(kv => kv.Key))) { + yield return fullName; + } + + List OrderImportNames(IEnumerable fullNames) { + return fullNames.OrderBy(n => n, ImportNameComparer.Instance).ToList(); + } + + bool FilterSimpleName(KeyValuePair kv) => kv.Key.IndexOf(".") < 0; + bool FilterImportedMembers(KeyValuePair kv) => !kv.Value.IsModule && kv.Value.MemberImported; + bool FilterImportedModuleMembers(KeyValuePair kv) => !kv.Value.IsModule && kv.Value.ModuleImported; + bool FilterImportedModules(KeyValuePair kv) => kv.Value.IsModule && kv.Value.MemberImported; + + bool FilterSourceDeclarations(IGrouping> group) { + var count = 0; + foreach (var entry in group) { + if (count++ > 0) { + return false; + } + + var value = entry.Value; + if (value.ModuleImported || value.ModuleImported) { + return false; + } + } + + return true; + } + } + + private static async Task AddCandidatesFromIndexAsync(IDocumentAnalysis analysis, + string name, + Dictionary importFullNameMap, + CancellationToken cancellationToken) { + var indexManager = analysis.ExpressionEvaluator.Services.GetService(); + if (indexManager == null) { + // indexing is not supported + return; + } + + var symbolsIncludingName = await indexManager.WorkspaceSymbolsAsync(name, maxLength: int.MaxValue, includeLibraries: true, cancellationToken); + + // we only consider exact matches rather than partial matches + var symbolsWithName = symbolsIncludingName.Where(Include); + + var analyzer = analysis.ExpressionEvaluator.Services.GetService(); + var pathResolver = analysis.Document.Interpreter.ModuleResolution.CurrentPathResolver; + + var modules = ImmutableArray.Empty; + foreach (var symbolAndModuleName in symbolsWithName.Select(s => (symbol: s, moduleName: pathResolver.GetModuleNameByPath(s.DocumentPath)))) { + cancellationToken.ThrowIfCancellationRequested(); + + var key = $"{symbolAndModuleName.moduleName}.{symbolAndModuleName.symbol.Name}"; + var symbol = symbolAndModuleName.symbol; + + importFullNameMap.TryGetValue(key, out var existing); + + // we don't actually know whether this is a module. all we know is it appeared at + // Import statement. but most likely module, so we mark it as module for now. + // later when we check loaded module, if this happen to be loaded, this will get + // updated with more accurate data. + // if there happen to be multiple symbols with same name, we refer to mark it as module + var isModule = symbol.Kind == Indexing.SymbolKind.Module || existing.IsModule; + + // any symbol marked "Module" by indexer is imported. + importFullNameMap[key] = new ImportInfo( + moduleImported: isModule, + memberImported: isModule, + isModule); + } + + bool Include(FlatSymbol symbol) { + // we only suggest symbols that exist in __all__ + // otherwise, we show gigantic list from index + return symbol._existInAllVariable && + symbol.ContainerName == null && + CheckKind(symbol.Kind) && + symbol.Name == name; + } + + bool CheckKind(Indexing.SymbolKind kind) { + switch (kind) { + case Indexing.SymbolKind.Module: + case Indexing.SymbolKind.Namespace: + case Indexing.SymbolKind.Package: + case Indexing.SymbolKind.Class: + case Indexing.SymbolKind.Enum: + case Indexing.SymbolKind.Interface: + case Indexing.SymbolKind.Function: + case Indexing.SymbolKind.Constant: + case Indexing.SymbolKind.Struct: + return true; + default: + return false; + } + } + } + + private CodeAction CreateCodeAction(IDocumentAnalysis analysis, + Node node, + string moduleFullName, + Diagnostic[] diagnostics, + bool locallyInserted, + CancellationToken cancellationToken) { + var insertionPoint = GetInsertionInfo(analysis, node, moduleFullName, locallyInserted, cancellationToken); + if (insertionPoint == null) { + return null; + } + + var insertionText = insertionPoint.Value.InsertionText; + var titleText = locallyInserted ? Resources.ImportLocally.FormatUI(insertionText) : insertionText; + + var sb = new StringBuilder(); + sb.AppendIf(insertionPoint.Value.Range.start == insertionPoint.Value.Range.end, insertionPoint.Value.Indentation); + sb.Append(insertionPoint.Value.AddBlankLine ? insertionText + Environment.NewLine : insertionText); + sb.AppendIf(insertionPoint.Value.Range.start == insertionPoint.Value.Range.end, Environment.NewLine); + + var textEdits = new List(); + textEdits.Add(new TextEdit() { range = insertionPoint.Value.Range, newText = sb.ToString() }); + + if (insertionPoint.Value.AbbreviationOpt != null) { + textEdits.Add(new TextEdit() { range = node.GetSpan(analysis.Ast), newText = insertionPoint.Value.AbbreviationOpt }); + } + + var changes = new Dictionary { { analysis.Document.Uri, textEdits.ToArray() } }; + return new CodeAction() { title = titleText, kind = CodeActionKind.QuickFix, diagnostics = diagnostics, edit = new WorkspaceEdit() { changes = changes } }; + } + + private InsertionInfo? GetInsertionInfo(IDocumentAnalysis analysis, + Node node, + string fullyQualifiedName, + bool locallyInserted, + CancellationToken cancellationToken) { + var (body, indentation) = GetStartingPoint(analysis, node, locallyInserted, cancellationToken); + if (body == null) { + // no insertion point + return null; + } + + var importNodes = body.GetChildNodes().Where(c => c is ImportStatement || c is FromImportStatement).ToList(); + var lastImportNode = importNodes.LastOrDefault(); + + var abbreviation = GetAbbreviationForWellKnownModules(analysis, fullyQualifiedName); + + // first check whether module name is dotted or not + var dotIndex = fullyQualifiedName.LastIndexOf('.'); + if (dotIndex < 0) { + // there can't be existing import since we have the error + return new InsertionInfo(addBlankLine: lastImportNode == null, + GetInsertionText($"import {fullyQualifiedName}", abbreviation), + GetRange(analysis.Ast, body, lastImportNode), + indentation, + abbreviation); + } + + // see whether there is existing from * import * statement. + var fromPart = fullyQualifiedName.Substring(startIndex: 0, dotIndex); + var nameToAdd = fullyQualifiedName.Substring(dotIndex + 1); + foreach (var current in importNodes.Reverse().OfType()) { + if (current.Root.MakeString() == fromPart) { + return new InsertionInfo(addBlankLine: false, + GetInsertionText(current, fromPart, nameToAdd, abbreviation), + current.GetSpan(analysis.Ast), + indentation, + abbreviation); + } + } + + // add new from * import * statement + return new InsertionInfo(addBlankLine: lastImportNode == null, + GetInsertionText($"from {fromPart} import {nameToAdd}", abbreviation), + GetRange(analysis.Ast, body, lastImportNode), + indentation, + abbreviation); + } + + private static string GetAbbreviationForWellKnownModules(IDocumentAnalysis analysis, string fullyQualifiedName) { + if (WellKnownAbbreviationMap.TryGetValue(fullyQualifiedName, out var abbreviation)) { + // for now, use module wide unique name for abbreviation. even though technically we could use + // context based unique name since variable declared in lower scope will hide it and there is no conflict + return UniqueNameGenerator.Generate(analysis, abbreviation); + } + + return null; + } + + private static string GetInsertionText(string insertionText, string abbreviation) => + abbreviation == null ? insertionText : $"{insertionText} as {abbreviation}"; + + private string GetInsertionText(FromImportStatement fromImportStatement, string rootModuleName, string moduleNameToAdd, string abbreviation) { + var imports = fromImportStatement.Names.Select(n => n.Name) + .Concat(new string[] { GetInsertionText(moduleNameToAdd, abbreviation) }) + .OrderBy(n => n).ToList(); + + return $"from {rootModuleName} import {string.Join(", ", imports)}"; + } + + private Range GetRange(PythonAst ast, Statement body, Node lastImportNode) { + var position = GetPosition(ast, body, lastImportNode); + return new Range() { start = position, end = position }; + } + + private Position GetPosition(PythonAst ast, Statement body, Node lastImportNode) { + if (lastImportNode != null) { + var endLocation = lastImportNode.GetEnd(ast); + return new Position { line = endLocation.Line, character = 0 }; + } + + // firstNode must exist in this context + var firstNode = body.GetChildNodes().First(); + return new Position() { line = firstNode.GetStart(ast).Line - 1, character = 0 }; + } + + private (Statement body, string indentation) GetStartingPoint(IDocumentAnalysis analysis, + Node node, + bool locallyInserted, + CancellationToken cancellationToken) { + if (!locallyInserted) { + return (analysis.Ast.Body, string.Empty); + } + + var candidate = GetAncestorsOrThis(analysis.Ast.Body, node, cancellationToken).Where(p => p is FunctionDefinition).LastOrDefault(); + + // for now, only stop at FunctionDefinition. + // we can expand it to more scope if we want but this seems what other tool also provide as well. + // this will return closest scope from given node + switch (candidate) { + case FunctionDefinition functionDefinition: + return (functionDefinition.Body, GetIndentation(analysis.Ast, functionDefinition.Body)); + default: + // no local scope + return default; + } + } + + private string GetIndentation(PythonAst ast, Statement body) { + // first token must exist in current context + var firstToken = body.GetChildNodes().First(); + + // not sure how to handle a case where user is using "tab" instead of "space" + // for indentation. where can one get tab over indentation option? + return new string(' ', firstToken.GetStart(ast).Column - 1); + } + + private List GetAncestorsOrThis(Node root, Node node, CancellationToken cancellationToken) { + var parentChain = new List(); + + // there seems no way to go up the parent chain. always has to go down from the top + while (root != null) { + cancellationToken.ThrowIfCancellationRequested(); + + var temp = root; + root = null; + + // this assumes node is not overlapped and children are ordered from left to right + // in textual position + foreach (var current in temp.GetChildNodes()) { + if (!current.IndexSpan.Contains(node.IndexSpan)) { + continue; + } + + parentChain.Add(current); + root = current; + break; + } + } + + return parentChain; + } + + private void CollectCandidates(ModuleInfo moduleInfo, + string name, + Dictionary importFullNameMap, + CancellationToken cancellationToken) { + if (!moduleInfo.CheckCircularImports()) { + // bail out on circular imports + return; + } + + // add non module (imported) member + AddNonImportedMemberWithName(moduleInfo, name, importFullNameMap); + + // add module (imported) members if it shows up in __all__ + // + // we are doing recursive dig down rather than just going through all modules loaded linearly + // since path to how to get to a module is important. + // for example, "join" is defined in "ntpath" or "macpath" and etc, but users are supposed to + // use it through "os.path" which will automatically point to right module ex, "ntpath" based on + // environment rather than "ntpath" directly. if we just go through module in flat list, then + // we can miss "os.path" since it won't show in the module list. + // for these modules that are supposed to be used with indirect path (imported name of the module), + // we need to dig down to collect those with right path. + foreach (var memberName in GetAllVariables(moduleInfo.Analysis)) { + cancellationToken.ThrowIfCancellationRequested(); + + var pythonModule = moduleInfo.Module.GetMember(memberName) as IPythonModule; + if (pythonModule == null) { + continue; + } + + var fullName = $"{moduleInfo.FullName}.{memberName}"; + if (string.Equals(memberName, name)) { + // nested module are all imported + AddNameParts(fullName, moduleImported: true, memberImported: true, pythonModule, importFullNameMap); + } + + // make sure we dig down modules only if we can use it from imports + // for example, user can do "from numpy import char" to import char [defchararray] module + // but user can not do "from numpy.char import x" since it is not one of known modules to us. + // in contrast, users can do "from os import path" to import path [ntpath] module + // but also can do "from os.path import x" since "os.path" is one of known moudles to us. + var result = AstUtilities.FindImports( + moduleInfo.CurrentFileAnalysis.Document.Interpreter.ModuleResolution.CurrentPathResolver, + moduleInfo.CurrentFileAnalysis.Document.FilePath, + GetRootNames(fullName), + dotCount: 0, + forceAbsolute: true); + + if (result is ImportNotFound) { + continue; + } + + moduleInfo.AddName(memberName); + CollectCandidates(moduleInfo.With(pythonModule), name, importFullNameMap, cancellationToken); + moduleInfo.PopName(); + } + + // pop this module out so we can get to this module from + // different path. + // ex) A -> B -> [C] and A -> D -> [C] + moduleInfo.ForgetModule(); + } + + private IEnumerable GetRootNames(string fullName) { + return fullName.Split('.'); + } + + private void AddNonImportedMemberWithName(ModuleInfo moduleInfo, string name, Dictionary importFullNameMap) { + // for now, skip any protected or private member + if (name.StartsWith("_")) { + return; + } + + var pythonType = moduleInfo.Module.GetMember(name); + if (pythonType == null || pythonType is IPythonModule || pythonType.IsUnknown()) { + return; + } + + // skip any imported member (non module member) unless it is explicitly on __all__ + if (moduleInfo.Analysis.GlobalScope.Imported.TryGetVariable(name, out var importedVariable) && + object.Equals(pythonType, importedVariable.Value) && + GetAllVariables(moduleInfo.Analysis).All(s => !string.Equals(s, name))) { + return; + } + + moduleInfo.AddName(name); + AddNameParts(moduleInfo.FullName, moduleInfo.ModuleImported, importedVariable != null, pythonType, importFullNameMap); + moduleInfo.PopName(); + } + + private static void AddNameParts( + string fullName, bool moduleImported, bool memberImported, IPythonType symbol, Dictionary moduleFullNameMap) { + // one of case this can happen is if module's fullname is "a.b.c" and module "a.b" also import module "a.b.c" as "c" making + // fullname same "a.b.c". in this case, we mark it as "imported" since we refer one explicily shown in "__all__" to show + // higher rank than others + if (moduleFullNameMap.TryGetValue(fullName, out var info)) { + moduleImported |= info.ModuleImported; + } + + moduleFullNameMap[fullName] = new ImportInfo(moduleImported, memberImported, symbol); + } + + private IEnumerable GetAllVariables(IDocumentAnalysis analysis) { + if (analysis?.GlobalScope == null) { + return Array.Empty(); + } + + // this is different than StartImportMemberNames since that only returns something when + // all entries are known. for import, we are fine doing best effort + if (analysis.GlobalScope.Variables.TryGetVariable("__all__", out var variable) && + variable?.Value is IPythonCollection collection) { + return collection.Contents + .OfType() + .Select(c => c.GetString()) + .Where(s => !string.IsNullOrEmpty(s)); + } + + return Array.Empty(); + } + + private class ImportNameComparer : IComparer { + public static readonly ImportNameComparer Instance = new ImportNameComparer(); + + private ImportNameComparer() { } + + public int Compare(string x, string y) { + const string underscore = "_"; + + // move "_" to back of the list + if (x.StartsWith(underscore) && y.StartsWith(underscore)) { + return x.CompareTo(y); + } + if (x.StartsWith(underscore)) { + return 1; + } + if (y.StartsWith(underscore)) { + return -1; + } + + return x.CompareTo(y); + } + } + + private struct InsertionInfo { + public readonly bool AddBlankLine; + public readonly string InsertionText; + public readonly Range Range; + public readonly string Indentation; + public readonly string AbbreviationOpt; + + public InsertionInfo(bool addBlankLine, string insertionText, Range range, string indentation, string abbreviationOpt = null) { + AddBlankLine = addBlankLine; + InsertionText = insertionText; + Range = range; + Indentation = indentation; + AbbreviationOpt = abbreviationOpt; + } + } + + private struct Input { + public readonly Node Context; + public readonly string Identifier; + public readonly string ModuleFullNameOpt; + + public Input(Node context, string identifier, string moduleFullNameOpt = null) { + Context = context; + Identifier = identifier; + ModuleFullNameOpt = moduleFullNameOpt; + } + } + + private struct ModuleInfo { + public readonly IDocumentAnalysis CurrentFileAnalysis; + public readonly IPythonModule Module; + public readonly List NameParts; + public readonly bool ModuleImported; + + private readonly HashSet _visited; + + public IDocumentAnalysis Analysis => Module.Analysis; + public string FullName => string.Join('.', NameParts); + + public ModuleInfo(IDocumentAnalysis document) : + this(document, module: null, new List(), moduleImported: false) { + } + + private ModuleInfo(IDocumentAnalysis document, IPythonModule module, List nameParts, bool moduleImported) : + this() { + CurrentFileAnalysis = document; + Module = module; + NameParts = nameParts; + ModuleImported = moduleImported; + + _visited = new HashSet(); + } + + public bool CheckCircularImports() => Module != null && _visited.Add(Module); + public void ForgetModule() => _visited.Remove(Module); + + public void AddName(string memberName) => NameParts.Add(memberName); + public void PopName() => NameParts.RemoveAt(NameParts.Count - 1); + + public ModuleInfo With(IPythonModule module) { + return new ModuleInfo(CurrentFileAnalysis, module, NameParts, moduleImported: true); + } + + public ModuleInfo Reset(IPythonModule module) { + Debug.Assert(_visited.Count == 0); + + NameParts.Clear(); + NameParts.Add(module.Name); + + return new ModuleInfo(CurrentFileAnalysis, module, NameParts, moduleImported: false); + } + } + + [DebuggerDisplay("{Symbol?.Name} Module:{IsModule} ({ModuleImported} {MemberImported})")] + private struct ImportInfo { + // only one that shows up in "__all__" will be imported + // containing module is imported + public readonly bool ModuleImported; + // containing symbol is imported + public readonly bool MemberImported; + + public readonly bool IsModule; + public readonly IPythonType Symbol; + + public ImportInfo(bool moduleImported, bool memberImported, IPythonType symbol) : + this(moduleImported, memberImported, symbol.MemberType == PythonMemberType.Module) { + Symbol = symbol; + } + + public ImportInfo(bool moduleImported, bool memberImported, bool isModule) { + ModuleImported = moduleImported; + MemberImported = memberImported; + IsModule = isModule; + Symbol = null; + } + } + } +} + diff --git a/src/LanguageServer/Impl/Completion/CompletionContext.cs b/src/LanguageServer/Impl/Completion/CompletionContext.cs index b40f8a44b..f71f5b944 100644 --- a/src/LanguageServer/Impl/Completion/CompletionContext.cs +++ b/src/LanguageServer/Impl/Completion/CompletionContext.cs @@ -15,6 +15,7 @@ using System.Linq; using Microsoft.Python.Analysis; +using Microsoft.Python.Core; using Microsoft.Python.Core.Text; using Microsoft.Python.Parsing.Ast; @@ -28,12 +29,14 @@ internal sealed class CompletionContext { public int Position { get; } public TokenSource TokenSource => _ts ?? (_ts = new TokenSource(Analysis.Document, Position)); public CompletionItemSource ItemSource { get; } + public IServiceContainer Services { get; } - public CompletionContext(IDocumentAnalysis analysis, SourceLocation location, CompletionItemSource itemSource) { + public CompletionContext(IDocumentAnalysis analysis, SourceLocation location, CompletionItemSource itemSource, IServiceContainer services) { Location = location; Analysis = analysis; Position = Ast.LocationToIndex(location); ItemSource = itemSource; + Services = services; } public SourceLocation IndexToLocation(int index) => Ast.IndexToLocation(index); diff --git a/src/LanguageServer/Impl/Completion/CompletionItemSource.cs b/src/LanguageServer/Impl/Completion/CompletionItemSource.cs index 8e1fbc6ee..1cf2847f8 100644 --- a/src/LanguageServer/Impl/Completion/CompletionItemSource.cs +++ b/src/LanguageServer/Impl/Completion/CompletionItemSource.cs @@ -56,7 +56,7 @@ public CompletionItemEx CreateCompletionItem(string text, CompletionItemKind kin // Place regular items first, advanced entries last sortText = char.IsLetter(text, 0) ? "1" : "2", kind = kind, - documentation = !t.IsUnknown() ? _docSource.GetHover(label ?? text, member, self) : null, + documentation = !t.IsUnknown() ? _docSource.GetHover(label ?? text, member, self, true) : null, // Custom fields used by the LS extensions that may modify // the completion list. Not passed to the client. Member = member, diff --git a/src/LanguageServer/Impl/Completion/CompletionSource.cs b/src/LanguageServer/Impl/Completion/CompletionSource.cs index 7f0a3f714..d8666c5a5 100644 --- a/src/LanguageServer/Impl/Completion/CompletionSource.cs +++ b/src/LanguageServer/Impl/Completion/CompletionSource.cs @@ -17,6 +17,7 @@ using Microsoft.Python.Analysis; using Microsoft.Python.Analysis.Analyzer.Expressions; using Microsoft.Python.Analysis.Modules; +using Microsoft.Python.Core; using Microsoft.Python.Core.Text; using Microsoft.Python.Parsing; using Microsoft.Python.Parsing.Ast; @@ -24,9 +25,11 @@ namespace Microsoft.Python.LanguageServer.Completion { internal sealed class CompletionSource { private readonly CompletionItemSource _itemSource; + private readonly IServiceContainer _services; - public CompletionSource(IDocumentationSource docSource, ServerSettings.PythonCompletionOptions completionSettings) { + public CompletionSource(IDocumentationSource docSource, ServerSettings.PythonCompletionOptions completionSettings, IServiceContainer services) { _itemSource = new CompletionItemSource(docSource, completionSettings); + _services = services; } public ServerSettings.PythonCompletionOptions Options { @@ -39,7 +42,7 @@ public CompletionResult GetCompletions(IDocumentAnalysis analysis, SourceLocatio return CompletionResult.Empty; } - var context = new CompletionContext(analysis, location, _itemSource); + var context = new CompletionContext(analysis, location, _itemSource, _services); ExpressionLocator.FindExpression(analysis.Ast, location, FindExpressionOptions.Complete, out var expression, out var statement, out var scope); diff --git a/src/LanguageServer/Impl/Completion/ExpressionCompletion.cs b/src/LanguageServer/Impl/Completion/ExpressionCompletion.cs index 5e9ff8d33..83703179c 100644 --- a/src/LanguageServer/Impl/Completion/ExpressionCompletion.cs +++ b/src/LanguageServer/Impl/Completion/ExpressionCompletion.cs @@ -13,13 +13,15 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System.Collections.Generic; +using System.IO; +using System.Linq; using Microsoft.Python.Analysis; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Values; +using Microsoft.Python.Core.IO; using Microsoft.Python.LanguageServer.Protocol; using Microsoft.Python.Parsing.Ast; -using System.Collections.Generic; -using System.Linq; namespace Microsoft.Python.LanguageServer.Completion { internal static class ExpressionCompletion { @@ -41,19 +43,12 @@ private static IEnumerable GetItemsFromExpression(Expression e, if (!value.IsUnknown()) { var type = value.GetPythonType(); - if(type is IPythonClassType cls) { + if (type is IPythonClassType cls) { return GetClassItems(cls, e, context); } - var items = new List(); - foreach (var t in type.GetMemberNames().ToArray()) { - var m = type.GetMember(t); - if (m is IVariable v && v.Source != VariableSource.Declaration) { - continue; - } - items.Add(context.ItemSource.CreateCompletionItem(t, m, type)); - } - return items; + return type.GetMemberNames() + .Select(name => context.ItemSource.CreateCompletionItem(name, type.GetMember(name), type)); } return Enumerable.Empty(); } @@ -63,8 +58,8 @@ private static IEnumerable GetClassItems(IPythonClassType cls, E // See if we are completing on self. Note that we may be inside inner function // that does not necessarily have 'self' argument so we are looking beyond local // scope. We then check that variable type matches the class type, if any. - var selfVariable = eval.LookupNameInScopes("self"); - var completingOnSelf = cls.Equals(selfVariable?.GetPythonType()) && e is NameExpression nex && nex.Name == "self"; + var classVariable = eval.LookupNameInScopes("self") ?? eval.LookupNameInScopes("cls"); + var completingOnClass = cls.Equals(classVariable?.GetPythonType()) && e is NameExpression nex && (nex.Name == "self" || nex.Name == "cls"); var items = new List(); var names = cls.GetMemberNames().ToArray(); @@ -79,7 +74,7 @@ private static IEnumerable GetClassItems(IPythonClassType cls, E var unmangledName = cls.UnmangleMemberName(t); if (!string.IsNullOrEmpty(unmangledName)) { // Hide private variables outside of the class scope. - if (!completingOnSelf && cls.IsPrivateMember(t)) { + if (!completingOnClass && cls.IsPrivateMember(t)) { continue; } items.Add(context.ItemSource.CreateCompletionItem(unmangledName, m, cls)); diff --git a/src/LanguageServer/Impl/Completion/FunctionDefinitionCompletion.cs b/src/LanguageServer/Impl/Completion/FunctionDefinitionCompletion.cs index 173173f1e..7fef36755 100644 --- a/src/LanguageServer/Impl/Completion/FunctionDefinitionCompletion.cs +++ b/src/LanguageServer/Impl/Completion/FunctionDefinitionCompletion.cs @@ -72,7 +72,34 @@ private static string MakeOverrideCompletionString(string indentation, IPythonFu var fn = overload.ClassMember as IPythonFunctionType; var skipFirstParameters = fn?.IsStatic == true ? overload.Parameters : overload.Parameters.Skip(1); - sb.AppendLine(overload.Name + "(" + string.Join(", ", overload.Parameters.Select(p => MakeOverrideParameter(p, p.DefaultValueString))) + "):"); + var addComma = false; + var addMarker = false; + + sb.Append(overload.Name); + sb.Append('('); + + foreach (var p in overload.Parameters) { + if (addComma) { + sb.Append(", "); + } else { + addComma = true; + } + + if (p.Kind == ParameterKind.PositionalOnly) { + addMarker = true; + } else if (addMarker && p.Kind != ParameterKind.PositionalOnly) { + sb.Append("/, "); + addMarker = false; + } + + sb.Append(MakeOverrideParameter(p, p.DefaultValueString)); + } + + if (addMarker) { + sb.Append(", /"); + } + + sb.AppendLine("):"); sb.Append(indentation); if (overload.Parameters.Count > 0) { diff --git a/src/LanguageServer/Impl/Completion/ImportCompletion.cs b/src/LanguageServer/Impl/Completion/ImportCompletion.cs index 9fb15e009..00f3f5b4a 100644 --- a/src/LanguageServer/Impl/Completion/ImportCompletion.cs +++ b/src/LanguageServer/Impl/Completion/ImportCompletion.cs @@ -15,13 +15,14 @@ using System; using System.Collections.Generic; -using System.IO; using System.Linq; using Microsoft.Python.Analysis.Core.DependencyResolution; +using Microsoft.Python.Analysis.Core.Interpreter; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Core; using Microsoft.Python.Core.Text; using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.Parsing; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.LanguageServer.Completion { @@ -41,7 +42,7 @@ public static CompletionResult TryGetCompletions(ImportStatement import, Complet if (name != null && context.Position >= name.StartIndex) { if (context.Position > name.EndIndex && name.EndIndex > name.StartIndex) { var applicableSpan = context.GetApplicableSpanFromLastToken(import); - return new CompletionResult(new []{ CompletionItemSource.AsKeyword }, applicableSpan); + return new CompletionResult(new[] { CompletionItemSource.AsKeyword }, applicableSpan); } if (name.Names.Count == 0 || name.Names[0].EndIndex >= context.Position) { @@ -52,7 +53,7 @@ public static CompletionResult TryGetCompletions(ImportStatement import, Complet var mres = document.Interpreter.ModuleResolution; var names = name.Names.TakeWhile(n => n.EndIndex < context.Position).Select(n => n.Name); var importSearchResult = mres.CurrentPathResolver.GetImportsFromAbsoluteName(document.FilePath, names, import.ForceAbsolute); - return GetResultFromImportSearch(importSearchResult, context, false); + return GetResultFromImportSearch(importSearchResult, context, false, modulesOnly: true); } } return null; @@ -60,7 +61,7 @@ public static CompletionResult TryGetCompletions(ImportStatement import, Complet public static CompletionResult GetCompletionsInFromImport(FromImportStatement fromImport, CompletionContext context) { // No more completions after '*', ever! - if (fromImport.Names != null && fromImport.Names.Any(n => n?.Name == "*" && context.Position > n.EndIndex)) { + if (fromImport.Names.Any(n => n?.Name == "*" && context.Position > n.EndIndex)) { return CompletionResult.Empty; } @@ -134,15 +135,17 @@ public static CompletionResult GetCompletionsInFromImport(FromImportStatement fr } private static IEnumerable GetAllImportableModules(CompletionContext context) { - var mres = context.Analysis.Document.Interpreter.ModuleResolution; - var modules = mres.CurrentPathResolver.GetAllModuleNames(); + var interpreter = context.Analysis.Document.Interpreter; + var languageVersion = interpreter.LanguageVersion.ToVersion(); + var includeImplicit = !ModulePath.PythonVersionRequiresInitPyFiles(languageVersion); + var modules = interpreter.ModuleResolution.CurrentPathResolver.GetAllImportableModuleNames(includeImplicit); return modules .Where(n => !string.IsNullOrEmpty(n)) .Distinct() .Select(n => CompletionItemSource.CreateCompletionItem(n, CompletionItemKind.Module)); } - private static CompletionResult GetResultFromImportSearch(IImportSearchResult importSearchResult, CompletionContext context, bool prependStar, SourceSpan? applicableSpan = null) { + private static CompletionResult GetResultFromImportSearch(IImportSearchResult importSearchResult, CompletionContext context, bool prependStar, SourceSpan? applicableSpan = null, bool modulesOnly = false) { var document = context.Analysis.Document; var mres = document.Interpreter.ModuleResolution; @@ -160,16 +163,19 @@ private static CompletionResult GetResultFromImportSearch(IImportSearchResult im default: return CompletionResult.Empty; } - + var completions = new List(); if (prependStar) { completions.Add(CompletionItemSource.Star); } + var memberNames = (module?.GetMemberNames().Where(n => !string.IsNullOrEmpty(n)) ?? Enumerable.Empty()).ToHashSet(); if (module != null) { - completions.AddRange(module.GetMemberNames() - .Where(n => !string.IsNullOrEmpty(n)) - .Select(n => context.ItemSource.CreateCompletionItem(n, module.GetMember(n)))); + var moduleMembers = memberNames + .Select(n => (n, m: module.GetMember(n))) + .Where(pair => !modulesOnly || pair.m is IPythonModule) + .Select(pair => context.ItemSource.CreateCompletionItem(pair.n, pair.m)); + completions.AddRange(moduleMembers); } if (importSearchResult is IImportChildrenSource children) { @@ -178,14 +184,19 @@ private static CompletionResult GetResultFromImportSearch(IImportSearchResult im continue; } + string name = null; switch (imports) { case ImplicitPackageImport packageImport: - completions.Add(CompletionItemSource.CreateCompletionItem(packageImport.Name, CompletionItemKind.Module)); + name = packageImport.Name; break; case ModuleImport moduleImport when !moduleImport.ModulePath.PathEquals(document.FilePath): - completions.Add(CompletionItemSource.CreateCompletionItem(moduleImport.Name, CompletionItemKind.Module)); + name = moduleImport.Name; break; } + + if (name != null && !memberNames.Contains(name)) { + completions.Add(CompletionItemSource.CreateCompletionItem(name, CompletionItemKind.Module)); + } } } diff --git a/src/LanguageServer/Impl/Completion/TopLevelCompletion.cs b/src/LanguageServer/Impl/Completion/TopLevelCompletion.cs index 6a1e735a4..64bcdd0d6 100644 --- a/src/LanguageServer/Impl/Completion/TopLevelCompletion.cs +++ b/src/LanguageServer/Impl/Completion/TopLevelCompletion.cs @@ -22,6 +22,7 @@ using Microsoft.Python.Core; using Microsoft.Python.Core.Text; using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.Parsing; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.LanguageServer.Completion { @@ -61,20 +62,24 @@ public static CompletionResult GetCompletions(Node statement, ScopeStatement sco // Add possible function arguments. var finder = new ExpressionFinder(context.Ast, new FindExpressionOptions { Calls = true }); if (finder.GetExpression(context.Position) is CallExpression callExpr && callExpr.GetArgumentAtIndex(context.Ast, context.Position, out _)) { - var value = eval.GetValueFromExpression(callExpr.Target); - if (value?.GetPythonType() is IPythonFunctionType ft) { - var arguments = ft.Overloads.SelectMany(o => o.Parameters).Select(p => p?.Name) - .Where(n => !string.IsNullOrEmpty(n)) - .Distinct() - .Except(callExpr.Args.MaybeEnumerate().Select(a => a.Name).Where(n => !string.IsNullOrEmpty(n))) - .Select(n => CompletionItemSource.CreateCompletionItem($"{n}=", CompletionItemKind.Variable)) - .ToArray(); - - items = items.Concat(arguments).ToArray(); + using (context.Analysis.ExpressionEvaluator.OpenScope(context.Analysis.Document, scopeStatement)) { + var value = eval.GetValueFromExpression(callExpr.Target); + var ft = value.TryGetFunctionType(); + if (ft != null) { + var arguments = ft.Overloads.SelectMany(o => o.Parameters).Select(p => p?.Name) + .Where(n => !string.IsNullOrEmpty(n)) + .Distinct() + .Except(callExpr.Args.MaybeEnumerate().Select(a => a.Name).Where(n => !string.IsNullOrEmpty(n))) + .Select(n => CompletionItemSource.CreateCompletionItem($"{n}=", CompletionItemKind.Variable)) + .ToArray(); + + items = items.Concat(arguments).ToArray(); + } } } - var keywords = GetKeywordItems(context, options, scopeStatement); + var keywords = GetKeywordItems(context, options, scopeStatement).ToArray(); + items = items.Where(item => !keywords.Any(i => i.insertText == item.insertText)); items = items.Concat(keywords); return new CompletionResult(items, applicableSpan); diff --git a/src/LanguageServer/Impl/Definitions/IDocumentationSource.cs b/src/LanguageServer/Impl/Definitions/IDocumentationSource.cs index f4a7fd30b..75233c068 100644 --- a/src/LanguageServer/Impl/Definitions/IDocumentationSource.cs +++ b/src/LanguageServer/Impl/Definitions/IDocumentationSource.cs @@ -20,8 +20,8 @@ namespace Microsoft.Python.LanguageServer { public interface IDocumentationSource { InsertTextFormat DocumentationFormat { get; } - MarkupContent GetHover(string name, IMember member, IPythonType self = null); - string GetSignatureString(IPythonFunctionType ft, IPythonType self, out IndexSpan[] parameterSpans, int overloadIndex = 0, string name = null); + MarkupContent GetHover(string name, IMember member, IPythonType self = null, bool includeClassInit = false); + string GetSignatureString(IPythonFunctionType ft, IPythonType self, out (IndexSpan, IParameterInfo)[] parameterSpans, int overloadIndex = 0, string name = null, bool noReturn = false); MarkupContent FormatParameterDocumentation(IParameterInfo parameter); MarkupContent FormatDocumentation(string documentation); } diff --git a/src/LanguageServer/Impl/Definitions/IQuickFixCodeActionProvider.cs b/src/LanguageServer/Impl/Definitions/IQuickFixCodeActionProvider.cs new file mode 100644 index 000000000..299ea272e --- /dev/null +++ b/src/LanguageServer/Impl/Definitions/IQuickFixCodeActionProvider.cs @@ -0,0 +1,43 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis; +using Microsoft.Python.Analysis.Diagnostics; +using Microsoft.Python.Core.Collections; +using Microsoft.Python.LanguageServer.CodeActions; +using Microsoft.Python.LanguageServer.Protocol; + +namespace Microsoft.Python.LanguageServer { + public interface IQuickFixCodeActionProvider { + /// + /// Returns error code this code action can provide fix for. This error code must be same as ones that are reported to host as diagnostics + /// for example, error code from linter + /// + ImmutableArray FixableDiagnostics { get; } + + /// + /// Returns that can potentially fix given diagnostic + /// + /// of the file where reported + /// Settings related to code actions one can query to get user preferences + /// that code action is supposed to fix + /// + /// that can fix the given + Task> GetCodeActionsAsync(IDocumentAnalysis analysis, CodeActionSettings settings, DiagnosticsEntry diagnostic, CancellationToken cancellation); + } +} diff --git a/src/LanguageServer/Impl/Definitions/IRefactoringCodeActionProvider.cs b/src/LanguageServer/Impl/Definitions/IRefactoringCodeActionProvider.cs new file mode 100644 index 000000000..761534b35 --- /dev/null +++ b/src/LanguageServer/Impl/Definitions/IRefactoringCodeActionProvider.cs @@ -0,0 +1,36 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis; +using Microsoft.Python.Core.Text; +using Microsoft.Python.LanguageServer.CodeActions; +using Microsoft.Python.LanguageServer.Protocol; + +namespace Microsoft.Python.LanguageServer { + public interface IRefactoringCodeActionProvider { + /// + /// Returns for the given . What it would do is up to the refactoring + /// + /// of the file where exists + /// Settings related to code actions one can query to get user preferences + /// Range where refactoring is called upon + /// + /// that will update user code or context + Task> GetCodeActionsAsync(IDocumentAnalysis analysis, CodeActionSettings settings, Range range, CancellationToken cancellation); + } +} diff --git a/src/LanguageServer/Impl/Diagnostics/DiagnosticExtensions.cs b/src/LanguageServer/Impl/Diagnostics/DiagnosticExtensions.cs new file mode 100644 index 000000000..6a13be8cf --- /dev/null +++ b/src/LanguageServer/Impl/Diagnostics/DiagnosticExtensions.cs @@ -0,0 +1,45 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using Microsoft.Python.Analysis.Diagnostics; +using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.Parsing; + +namespace Microsoft.Python.LanguageServer.Diagnostics { + internal static class DiagnosticExtensions { + public static Diagnostic ToDiagnostic(this DiagnosticsEntry diagnostic, string source = "Python") { + return new Diagnostic { + range = diagnostic.SourceSpan, + severity = diagnostic.Severity.ToDiagnosticSeverity(), + source = source, + code = diagnostic.ErrorCode, + message = diagnostic.Message, + }; + } + + public static DiagnosticSeverity ToDiagnosticSeverity(this Severity severity) { + switch (severity) { + case Severity.Warning: + return DiagnosticSeverity.Warning; + case Severity.Information: + return DiagnosticSeverity.Information; + case Severity.Hint: + return DiagnosticSeverity.Hint; + default: + return DiagnosticSeverity.Error; + } + } + } +} diff --git a/src/LanguageServer/Impl/Diagnostics/DiagnosticsService.cs b/src/LanguageServer/Impl/Diagnostics/DiagnosticsService.cs index fd28ecae7..3fd041b16 100644 --- a/src/LanguageServer/Impl/Diagnostics/DiagnosticsService.cs +++ b/src/LanguageServer/Impl/Diagnostics/DiagnosticsService.cs @@ -40,7 +40,10 @@ public void Clear(DiagnosticSource source) { _entries[source] = Array.Empty(); } - public void ClearAll() => _entries.Clear(); + public void ClearAll() { + Changed = Entries.Count > 0; + _entries.Clear(); + } public void SetDiagnostics(DiagnosticSource source, IReadOnlyList entries) { if (_entries.TryGetValue(source, out var existing)) { @@ -106,16 +109,7 @@ public void Replace(Uri documentUri, IEnumerable entries, Diag } } - public void Remove(Uri documentUri) { - lock (_lock) { - // Before removing the document, make sure we clear its diagnostics. - if (_diagnostics.TryGetValue(documentUri, out var d)) { - d.ClearAll(); - PublishDiagnostics(); - _diagnostics.Remove(documentUri); - } - } - } + public void Remove(Uri documentUri) => ClearDiagnostics(documentUri, true); public int PublishingDelay { get; set; } = 1000; @@ -151,18 +145,16 @@ private void OnIdle(object sender, EventArgs e) { private void PublishDiagnostics() { var diagnostics = new Dictionary(); lock (_lock) { - foreach (var d in _diagnostics) { - if (d.Value.Changed) { - diagnostics[d.Key] = d.Value; - d.Value.Changed = false; - } + foreach (var (uri, documentDiagnostics) in _diagnostics.Where(d => d.Value.Changed)) { + diagnostics[uri] = documentDiagnostics; + documentDiagnostics.Changed = false; } - foreach (var kvp in diagnostics) { + foreach (var (uri, documentDiagnostics) in diagnostics) { var parameters = new PublishDiagnosticsParams { - uri = kvp.Key, - diagnostics = Rdt.GetDocument(kvp.Key)?.IsOpen == true - ? FilterBySeverityMap(kvp.Value).Select(ToDiagnostic).ToArray() + uri = uri, + diagnostics = Rdt.GetDocument(uri)?.IsOpen == true + ? FilterBySeverityMap(documentDiagnostics).Select(d => d.ToDiagnostic()).ToArray() : Array.Empty() }; _clientApp.NotifyWithParameterObjectAsync("textDocument/publishDiagnostics", parameters).DoNotWait(); @@ -176,32 +168,6 @@ private void ClearAllDiagnostics() { } } - private static Diagnostic ToDiagnostic(DiagnosticsEntry e) { - DiagnosticSeverity s; - switch (e.Severity) { - case Severity.Warning: - s = DiagnosticSeverity.Warning; - break; - case Severity.Information: - s = DiagnosticSeverity.Information; - break; - case Severity.Hint: - s = DiagnosticSeverity.Hint; - break; - default: - s = DiagnosticSeverity.Error; - break; - } - - return new Diagnostic { - range = e.SourceSpan, - severity = s, - source = "Python", - code = e.ErrorCode, - message = e.Message, - }; - } - private IEnumerable FilterBySeverityMap(DocumentDiagnostics d) => d.Entries .SelectMany(kvp => kvp.Value) @@ -241,9 +207,19 @@ private void OnOpenDocument(object sender, DocumentEventArgs e) { private void OnCloseDocument(object sender, DocumentEventArgs e) => ClearDiagnostics(e.Document.Uri, false); private void OnRemoveDocument(object sender, DocumentEventArgs e) => ClearDiagnostics(e.Document.Uri, true); + /// + /// Removes document diagnostics (publishes empty set to the client). + /// If the document is still open, URI remains in the document diagnostics map. + /// + /// Document URI. + /// + /// True means the document is closed and its diagnostics should be + /// removed from the map (as opposed to document is still open and has no diagnostics). + /// private void ClearDiagnostics(Uri uri, bool remove) { lock (_lock) { - if (_diagnostics.TryGetValue(uri, out var _)) { + if (_diagnostics.TryGetValue(uri, out var d)) { + d.Changed = true; PublishDiagnostics(); if (remove) { _diagnostics.Remove(uri); diff --git a/src/LanguageServer/Impl/Documentation/DocstringConverter.cs b/src/LanguageServer/Impl/Documentation/DocstringConverter.cs index 046488540..3f345a11a 100644 --- a/src/LanguageServer/Impl/Documentation/DocstringConverter.cs +++ b/src/LanguageServer/Impl/Documentation/DocstringConverter.cs @@ -176,7 +176,7 @@ private static readonly (Regex, string)[] PotentialHeaders = new[] { private static readonly Regex TildaHeaderRegex = new Regex(@"^\s*~~~+$", RegexOptions.Singleline | RegexOptions.Compiled); private static readonly Regex PlusHeaderRegex = new Regex(@"^\s*\+\+\++$", RegexOptions.Singleline | RegexOptions.Compiled); private static readonly Regex LeadingAsteriskRegex = new Regex(@"^(\s+\* )(.*)$", RegexOptions.Singleline | RegexOptions.Compiled); - private static readonly Regex UnescapedMarkdownCharsRegex = new Regex(@"(?(); + var fs = Services.GetService(); + try { + var cachesRoot = Path.GetDirectoryName(mdc.CacheFolder); + foreach (var dir in fs + .GetFileSystemEntries(cachesRoot, $"{mdc.CacheFolderBaseName}*", SearchOption.TopDirectoryOnly) + .Where(e => fs.GetFileAttributes(e).HasFlag(FileAttributes.Directory))) { + fs.DeleteDirectory(dir, recursive: true); + } + } catch(IOException) { } catch(UnauthorizedAccessException) { } + } + } +} diff --git a/src/LanguageServer/Impl/Implementation/Server.Documents.cs b/src/LanguageServer/Impl/Implementation/Server.Documents.cs index 43b7937ec..37d8bdfcc 100644 --- a/src/LanguageServer/Impl/Implementation/Server.Documents.cs +++ b/src/LanguageServer/Impl/Implementation/Server.Documents.cs @@ -13,8 +13,6 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. -using System; -using System.Collections.Generic; using System.Diagnostics; using System.Linq; using Microsoft.Python.Analysis.Documents; diff --git a/src/LanguageServer/Impl/Implementation/Server.Editor.cs b/src/LanguageServer/Impl/Implementation/Server.Editor.cs index 0898c023c..46de921fa 100644 --- a/src/LanguageServer/Impl/Implementation/Server.Editor.cs +++ b/src/LanguageServer/Impl/Implementation/Server.Editor.cs @@ -14,6 +14,7 @@ // permissions and limitations under the License. using System; +using System.Collections.Generic; using System.Diagnostics; using System.Linq; using System.Threading; @@ -88,8 +89,11 @@ public async Task GotoDefinition(TextDocumentPositionParams @params _log?.Log(TraceEventType.Verbose, $"Goto Definition in {uri} at {@params.position}"); var analysis = await Document.GetAnalysisAsync(uri, Services, CompletionAnalysisTimeout, cancellationToken); - var reference = new DefinitionSource(Services).FindDefinition(analysis, @params.position, out _); - return reference != null ? new[] { reference } : Array.Empty(); + var ds = new DefinitionSource(Services); + var reference = ds.FindDefinition(analysis, @params.position, out _); + return reference != null && ds.CanNavigateToModule(reference.uri) + ? new[] { reference } + : Array.Empty(); } public async Task GotoDeclaration(TextDocumentPositionParams @params, CancellationToken cancellationToken) { @@ -97,8 +101,11 @@ public async Task GotoDeclaration(TextDocumentPositionParams @params, _log?.Log(TraceEventType.Verbose, $"Goto Declaration in {uri} at {@params.position}"); var analysis = await Document.GetAnalysisAsync(uri, Services, CompletionAnalysisTimeout, cancellationToken); - var reference = new DeclarationSource(Services).FindDefinition(analysis, @params.position, out _); - return reference != null ? new Location { uri = reference.uri, range = reference.range} : null; + var ds = new DeclarationSource(Services); + var reference = ds.FindDefinition(analysis, @params.position, out _); + return reference != null && ds.CanNavigateToModule(reference.uri) + ? new Location { uri = reference.uri, range = reference.range } + : null; } public Task FindReferences(ReferencesParams @params, CancellationToken cancellationToken) { @@ -107,10 +114,38 @@ public Task FindReferences(ReferencesParams @params, CancellationTo return new ReferenceSource(Services).FindAllReferencesAsync(uri, @params.position, ReferenceSearchOptions.All, cancellationToken); } + public Task DocumentHighlight(ReferencesParams @params, CancellationToken cancellationToken) { + var uri = @params.textDocument.uri; + _log?.Log(TraceEventType.Verbose, $"Document highlight in {uri} at {@params.position}"); + return new DocumentHighlightSource(Services).DocumentHighlightAsync(uri, @params.position, cancellationToken); + } + public Task Rename(RenameParams @params, CancellationToken cancellationToken) { var uri = @params.textDocument.uri; _log?.Log(TraceEventType.Verbose, $"Rename in {uri} at {@params.position}"); return new RenameSource(Services).RenameAsync(uri, @params.position, @params.newName, cancellationToken); } + + public async Task CodeAction(CodeActionParams @params, CancellationToken cancellationToken) { + var uri = @params.textDocument.uri; + _log?.Log(TraceEventType.Verbose, $"Code Action in {uri} at {@params.range}"); + + var codeActions = new List(); + var analysis = await Document.GetAnalysisAsync(uri, Services, CompletionAnalysisTimeout, cancellationToken); + + if (AskedFor(@params, CodeActionKind.Refactor)) { + codeActions.AddRange(await new RefactoringCodeActionSource(Services).GetCodeActionsAsync(analysis, _codeActionSettings, @params.range, cancellationToken)); + } + + if (@params.context.diagnostics?.Length > 0 && AskedFor(@params, CodeActionKind.QuickFix)) { + codeActions.AddRange(await new QuickFixCodeActionSource(Services).GetCodeActionsAsync(analysis, _codeActionSettings, @params.context.diagnostics, cancellationToken)); + } + + return codeActions.ToArray(); + + bool AskedFor(CodeActionParams p, string codeActionKind) { + return p.context.only == null || p.context.only.Any(s => s.StartsWith(codeActionKind)); + } + } } } diff --git a/src/LanguageServer/Impl/Implementation/Server.Telemetry.cs b/src/LanguageServer/Impl/Implementation/Server.Telemetry.cs index 4dcfdd079..b69a4a41e 100644 --- a/src/LanguageServer/Impl/Implementation/Server.Telemetry.cs +++ b/src/LanguageServer/Impl/Implementation/Server.Telemetry.cs @@ -13,6 +13,7 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System; using System.Diagnostics; using Microsoft.Python.Analysis.Analyzer; using Microsoft.Python.Core; @@ -29,28 +30,34 @@ private void OnAnalysisComplete(object sender, AnalysisCompleteEventArgs e) { return; } - double privateMB; - double peakPagedMB; - double workingMB; + try { + double privateMB; + double peakPagedMB; + double workingMB; - using (var proc = Process.GetCurrentProcess()) { - privateMB = proc.PrivateMemorySize64 / 1e+6; - peakPagedMB = proc.PeakPagedMemorySize64 / 1e+6; - workingMB = proc.WorkingSet64 / 1e+6; - } + using (var proc = Process.GetCurrentProcess()) { + privateMB = proc.PrivateMemorySize64 / 1e+6; + peakPagedMB = proc.PeakPagedMemorySize64 / 1e+6; + workingMB = proc.WorkingSet64 / 1e+6; + } - var te = new TelemetryEvent { - EventName = "python_language_server/analysis_complete", // TODO: Move this common prefix into Core. - }; + var te = new TelemetryEvent { + EventName = "python_language_server/analysis_complete", // TODO: Move this common prefix into Core. + }; - te.Measurements["privateMB"] = privateMB; - te.Measurements["peakPagedMB"] = peakPagedMB; - te.Measurements["workingMB"] = workingMB; - te.Measurements["elapsedMs"] = e.MillisecondsElapsed; - te.Measurements["moduleCount"] = e.ModuleCount; - te.Measurements["rdtCount"] = _rdt.DocumentCount; + te.Measurements["privateMB"] = privateMB; + te.Measurements["peakPagedMB"] = peakPagedMB; + te.Measurements["workingMB"] = workingMB; + te.Measurements["elapsedMs"] = e.MillisecondsElapsed; + te.Measurements["moduleCount"] = e.ModuleCount; + te.Measurements["rdtCount"] = _rdt.DocumentCount; - telemetry.SendTelemetryAsync(te).DoNotWait(); + telemetry.SendTelemetryAsync(te).DoNotWait(); + } catch(Exception ex) when (!ex.IsCriticalException()) { + // Workaround for https://github.com/microsoft/python-language-server/issues/1820 + // On some systems random DLL may get missing or otherwise not installed + // and we don't want to crash b/c of telemetry. + } } } } diff --git a/src/LanguageServer/Impl/Implementation/Server.cs b/src/LanguageServer/Impl/Implementation/Server.cs index 01c643ea9..b9e425d74 100644 --- a/src/LanguageServer/Impl/Implementation/Server.cs +++ b/src/LanguageServer/Impl/Implementation/Server.cs @@ -18,6 +18,7 @@ using System.Diagnostics; using System.Linq; using System.Reflection; +using System.Text; using System.Threading; using System.Threading.Tasks; using Microsoft.Python.Analysis; @@ -32,6 +33,7 @@ using Microsoft.Python.Core.IO; using Microsoft.Python.Core.Logging; using Microsoft.Python.Core.Services; +using Microsoft.Python.LanguageServer.CodeActions; using Microsoft.Python.LanguageServer.Completion; using Microsoft.Python.LanguageServer.Diagnostics; using Microsoft.Python.LanguageServer.Indexing; @@ -48,12 +50,15 @@ public sealed partial class Server : IDisposable { private IRunningDocumentTable _rdt; private ILogger _log; private IIndexManager _indexManager; + private PythonAnalyzer _analyzer; private InitializeParams _initParams; + private bool _initialized; private bool _watchSearchPaths; private PathsWatcher _pathsWatcher; private string[] _searchPaths; + private CodeActionSettings _codeActionSettings; public string Root { get; private set; } @@ -77,29 +82,33 @@ public Server(IServiceManager services) { public void Dispose() => _disposableBag.TryDispose(); #region Client message handling - private InitializeResult GetInitializeResult() => new InitializeResult { - capabilities = new ServerCapabilities { - textDocumentSync = new TextDocumentSyncOptions { - openClose = true, - change = TextDocumentSyncKind.Incremental - }, - completionProvider = new CompletionOptions { - triggerCharacters = new[] { "." } - }, - hoverProvider = true, - signatureHelpProvider = new SignatureHelpOptions { triggerCharacters = new[] { "(", ",", ")" } }, - definitionProvider = true, - referencesProvider = true, - workspaceSymbolProvider = true, - documentSymbolProvider = true, - renameProvider = true, - declarationProvider = true, - documentOnTypeFormattingProvider = new DocumentOnTypeFormattingOptions { - firstTriggerCharacter = "\n", - moreTriggerCharacter = new[] { ";", ":" } - }, - } - }; + private InitializeResult GetInitializeResult() { + return new InitializeResult { + capabilities = new ServerCapabilities { + textDocumentSync = new TextDocumentSyncOptions { + openClose = true, + change = TextDocumentSyncKind.Incremental + }, + completionProvider = new CompletionOptions { + triggerCharacters = new[] { "." } + }, + hoverProvider = true, + signatureHelpProvider = new SignatureHelpOptions { triggerCharacters = new[] { "(", ",", ")" } }, + definitionProvider = true, + referencesProvider = true, + workspaceSymbolProvider = true, + documentSymbolProvider = true, + documentHighlightProvider = true, + renameProvider = true, + declarationProvider = true, + documentOnTypeFormattingProvider = new DocumentOnTypeFormattingOptions { + firstTriggerCharacter = "\n", + moreTriggerCharacter = new[] { ";", ":" } + }, + codeActionProvider = new CodeActionOptions() { codeActionKinds = new string[] { CodeActionKind.QuickFix, CodeActionKind.Refactor } }, + } + }; + } public Task InitializeAsync(InitializeParams @params, CancellationToken cancellationToken = default) { _disposableBag.ThrowIfDisposed(); @@ -122,29 +131,36 @@ public async Task InitializedAsync(InitializedParams @params, CancellationToken _services.AddService(new DiagnosticsService(_services)); - var cacheFolderPath = initializationOptions?.cacheFolderPath; - var fs = _services.GetService(); - if (cacheFolderPath != null && !fs.DirectoryExists(cacheFolderPath)) { - _log?.Log(TraceEventType.Warning, Resources.Error_InvalidCachePath); - cacheFolderPath = null; - } + _analyzer = new PythonAnalyzer(_services); + _services.AddService(_analyzer); - var analyzer = new PythonAnalyzer(_services, cacheFolderPath); - _services.AddService(analyzer); - - analyzer.AnalysisComplete += OnAnalysisComplete; - _disposableBag.Add(() => analyzer.AnalysisComplete -= OnAnalysisComplete); + _analyzer.AnalysisComplete += OnAnalysisComplete; + _disposableBag.Add(() => _analyzer.AnalysisComplete -= OnAnalysisComplete); _services.AddService(new RunningDocumentTable(_services)); _rdt = _services.GetService(); - Version.TryParse(initializationOptions?.interpreter.properties?.Version, out var version); + var interpeterPath = initializationOptions?.interpreter.properties?.InterpreterPath; + Version version = null; + + if (!string.IsNullOrWhiteSpace(interpeterPath)) { + Version.TryParse(initializationOptions?.interpreter.properties?.Version, out version); + } else { + var fs = _services.GetService(); + var exePath = PathUtils.LookPath(fs, "python"); + if (exePath != null && TryGetPythonVersion(exePath, out version)) { + _log?.Log(TraceEventType.Information, Resources.UsingPythonFromPATH); + interpeterPath = exePath; + } else { + interpeterPath = null; + } + } var configuration = new InterpreterConfiguration( - interpreterPath: initializationOptions?.interpreter.properties?.InterpreterPath, + interpreterPath: interpeterPath, version: version ); - _services.AddService(new ModuleDatabase(_services)); + //_services.AddService(new ModuleDatabase(_services)); var typeshedPath = initializationOptions?.typeStubSearchPaths.FirstOrDefault(); userConfiguredPaths = userConfiguredPaths ?? initializationOptions?.searchPaths; @@ -160,7 +176,10 @@ public async Task InitializedAsync(InitializedParams @params, CancellationToken initializationOptions?.includeFiles, initializationOptions?.excludeFiles, _services.GetService()); + _indexManager.IndexWorkspace().DoNotWait(); + _analyzer.AnalysisComplete += IndexLibraries; + _disposableBag.Add(() => _analyzer.AnalysisComplete -= IndexLibraries); _services.AddService(_indexManager); _disposableBag.Add(_indexManager); @@ -168,7 +187,7 @@ public async Task InitializedAsync(InitializedParams @params, CancellationToken _completionSource = new CompletionSource( ChooseDocumentationSource(textDocCaps?.completion?.completionItem?.documentationFormat), - Settings.completion + Settings.completion, Services ); _hoverSource = new HoverSource( @@ -180,6 +199,14 @@ public async Task InitializedAsync(InitializedParams @params, CancellationToken ChooseDocumentationSource(sigInfo?.documentationFormat), sigInfo?.parameterInformation?.labelOffsetSupport == true ); + + _initialized = true; + } + + private void IndexLibraries(object o, AnalysisCompleteEventArgs e) { + _log?.Log(TraceEventType.Verbose, Resources.IndexingLibraries); + _indexManager.IndexSnapshot(_interpreter.ModuleResolution.CurrentPathResolver).DoNotWait(); + _analyzer.AnalysisComplete -= IndexLibraries; } public Task Shutdown() { @@ -190,12 +217,11 @@ public Task Shutdown() { public void DidChangeConfiguration(DidChangeConfigurationParams @params, CancellationToken cancellationToken) { _disposableBag.ThrowIfDisposed(); switch (@params.settings) { - case ServerSettings settings: { - Settings = settings; - _symbolHierarchyMaxSymbols = Settings.analysis.symbolsHierarchyMaxSymbols; - _completionSource.Options = Settings.completion; - break; - } + case ServerSettings settings: + Settings = settings; + _symbolHierarchyMaxSymbols = Settings.analysis.symbolsHierarchyMaxSymbols; + _completionSource.Options = Settings.completion; + break; default: _log?.Log(TraceEventType.Error, "change configuration notification sent unsupported settings"); break; @@ -227,6 +253,10 @@ public void HandleUserConfiguredPathsChange(ImmutableArray paths) { } } + public void HandleCodeActionsChange(CodeActionSettings settings) { + _codeActionSettings = settings; + } + #region Private Helpers private IDocumentationSource ChooseDocumentationSource(string[] kinds) { if (kinds == null) { @@ -256,6 +286,12 @@ private void ResetPathWatcher() { } private void ResetAnalyzer() { + if (!_initialized) { + // We haven't yet initialized everything, not even the builtins. + // Resetting here would break things. + return; + } + _log?.Log(TraceEventType.Information, Resources.ReloadingModules); _services.GetService().ResetAnalyzer().ContinueWith(t => { if (_watchSearchPaths) { @@ -266,6 +302,35 @@ private void ResetAnalyzer() { _log?.Log(TraceEventType.Information, Resources.AnalysisRestarted); }).DoNotWait(); } + + private bool TryGetPythonVersion(string exePath, out Version version) { + try { + using var process = new Process { + StartInfo = new ProcessStartInfo { + FileName = exePath, + Arguments = "-c \"import sys; print('{}.{}.{}'.format(*sys.version_info))\"", + UseShellExecute = false, + ErrorDialog = false, + CreateNoWindow = true, + StandardOutputEncoding = Encoding.UTF8, + RedirectStandardInput = true, + RedirectStandardOutput = true, + RedirectStandardError = true + } + }; + + process.ErrorDataReceived += (s, e) => { }; + process.Start(); + process.BeginErrorReadLine(); + + var output = process.StandardOutput.ReadToEnd(); + + return Version.TryParse(output, out version); + } catch (Exception ex) when (!ex.IsCriticalException()) { } + + version = null; + return false; + } #endregion } } diff --git a/src/LanguageServer/Impl/Indexing/AllVariableCollector.cs b/src/LanguageServer/Impl/Indexing/AllVariableCollector.cs new file mode 100644 index 000000000..0e1137816 --- /dev/null +++ b/src/LanguageServer/Impl/Indexing/AllVariableCollector.cs @@ -0,0 +1,107 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.LanguageServer.Indexing { + /// + /// This is a poor man's __all__ values collector. it uses only syntactic information to gather values. + /// + /// unlike the real one that actually binds expressions and + /// uses semantic data to build up __all__ information, this one's purpose is gathering cheap and fast but might be incorrect data + /// until more expensive analysis is done. + /// + internal class AllVariableCollector : PythonWalker { + private const string AllVariableName = "__all__"; + private readonly CancellationToken _cancellationToken; + + /// + /// names assigned to __all__ + /// + public readonly HashSet Names; + + public AllVariableCollector(CancellationToken cancellationToken) { + _cancellationToken = cancellationToken; + Names = new HashSet(); + } + + public override bool Walk(AssignmentStatement node) { + _cancellationToken.ThrowIfCancellationRequested(); + + // make sure we are dealing with __all__ assignment + if (node.Left.OfType().Any(n => n.Name == AllVariableName)) { + AddNames(node.Right as ListExpression); + } + + return base.Walk(node); + } + + public override bool Walk(AugmentedAssignStatement node) { + _cancellationToken.ThrowIfCancellationRequested(); + + if (node.Operator == Parsing.PythonOperator.Add && + node.Left is NameExpression nex && + nex.Name == AllVariableName) { + AddNames(node.Right as ListExpression); + } + + return base.Walk(node); + } + + public override bool Walk(CallExpression node) { + _cancellationToken.ThrowIfCancellationRequested(); + + if (node.Args.Count > 0 && + node.Target is MemberExpression me && + me.Target is NameExpression nex && + nex.Name == AllVariableName) { + var arg = node.Args[0].Expression; + + switch (me.Name) { + case "append": + AddName(arg); + break; + case "extend": + AddNames(arg as ListExpression); + break; + } + } + + return base.Walk(node); + } + + private void AddName(Expression item) { + if (item is ConstantExpression con && + con.Value is string name && + !string.IsNullOrEmpty(name)) { + Names.Add(name); + } + } + + private void AddNames(ListExpression list) { + // only support the form of __all__ = [ ... ] + if (list == null) { + return; + } + + foreach (var item in list.Items) { + AddName(item); + } + } + } +} diff --git a/src/LanguageServer/Impl/Indexing/IIndexManager.cs b/src/LanguageServer/Impl/Indexing/IIndexManager.cs index 9edaa18e0..a8947177e 100644 --- a/src/LanguageServer/Impl/Indexing/IIndexManager.cs +++ b/src/LanguageServer/Impl/Indexing/IIndexManager.cs @@ -17,16 +17,19 @@ using System.Collections.Generic; using System.Threading; using System.Threading.Tasks; +using Microsoft.Python.Analysis.Core.DependencyResolution; using Microsoft.Python.Analysis.Documents; namespace Microsoft.Python.LanguageServer.Indexing { internal interface IIndexManager : IDisposable { Task IndexWorkspace(CancellationToken ct = default); + Task IndexSnapshot(PathResolverSnapshot snapshot, CancellationToken ct = default); void ProcessNewFile(string path, IDocument doc); void ProcessClosedFile(string path); void ReIndexFile(string path, IDocument doc); void AddPendingDoc(IDocument doc); Task> HierarchicalDocumentSymbolsAsync(string path, CancellationToken cancellationToken = default); Task> WorkspaceSymbolsAsync(string query, int maxLength, CancellationToken cancellationToken = default); + Task> WorkspaceSymbolsAsync(string query, int maxLength, bool includeLibraries, CancellationToken cancellationToken = default); } } diff --git a/src/LanguageServer/Impl/Indexing/IndexManager.cs b/src/LanguageServer/Impl/Indexing/IndexManager.cs index d6ab967f8..6e054de50 100644 --- a/src/LanguageServer/Impl/Indexing/IndexManager.cs +++ b/src/LanguageServer/Impl/Indexing/IndexManager.cs @@ -16,9 +16,11 @@ using System; using System.Collections.Concurrent; using System.Collections.Generic; +using System.IO; using System.Linq; using System.Threading; using System.Threading.Tasks; +using Microsoft.Python.Analysis.Core.DependencyResolution; using Microsoft.Python.Analysis.Core.Interpreter; using Microsoft.Python.Analysis.Documents; using Microsoft.Python.Core.Diagnostics; @@ -30,13 +32,15 @@ namespace Microsoft.Python.LanguageServer.Indexing { internal class IndexManager : IIndexManager { private const int DefaultReIndexDelay = 350; - private readonly ISymbolIndex _symbolIndex; + private readonly PythonLanguageVersion _version; + private readonly ISymbolIndex _userCodeSymbolIndex; + private readonly ISymbolIndex _libraryCodeSymbolIndex; private readonly IFileSystem _fileSystem; private readonly string _workspaceRootPath; private readonly string[] _includeFiles; private readonly string[] _excludeFiles; private readonly DisposableBag _disposables = new DisposableBag(nameof(IndexManager)); - private readonly ConcurrentDictionary _pendingDocs = new ConcurrentDictionary(new UriDocumentComparer()); + private readonly ConcurrentDictionary _pendingDocs = new ConcurrentDictionary(UriDocumentComparer.Instance); private readonly DisposeToken _disposeToken = DisposeToken.Create(); public IndexManager(IFileSystem fileSystem, PythonLanguageVersion version, string rootPath, string[] includeFiles, @@ -46,16 +50,20 @@ public IndexManager(IFileSystem fileSystem, PythonLanguageVersion version, strin Check.ArgumentNotNull(nameof(excludeFiles), excludeFiles); Check.ArgumentNotNull(nameof(idleTimeService), idleTimeService); + _version = version; _fileSystem = fileSystem; _workspaceRootPath = rootPath; _includeFiles = includeFiles; _excludeFiles = excludeFiles; - _symbolIndex = new SymbolIndex(_fileSystem, version); + _userCodeSymbolIndex = new SymbolIndex(_fileSystem, version); + _libraryCodeSymbolIndex = new SymbolIndex(_fileSystem, version, libraryMode: true); + idleTimeService.Idle += OnIdle; _disposables - .Add(_symbolIndex) + .Add(_userCodeSymbolIndex) + .Add(_libraryCodeSymbolIndex) .Add(() => _disposeToken.TryMarkDisposed()) .Add(() => idleTimeService.Idle -= OnIdle); } @@ -66,15 +74,31 @@ public Task IndexWorkspace(CancellationToken ct = default) { var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(ct, _disposeToken.CancellationToken); var linkedCt = linkedCts.Token; return Task.Run(() => { - foreach (var fileInfo in WorkspaceFiles()) { - linkedCt.ThrowIfCancellationRequested(); - if (ModulePath.IsPythonSourceFile(fileInfo.FullName)) { - _symbolIndex.Parse(fileInfo.FullName); - } - } + var userFiles = WorkspaceFiles(); + CreateIndices(userFiles, _userCodeSymbolIndex, linkedCt); + }, linkedCt).ContinueWith(_ => linkedCts.Dispose()); + } + + public Task IndexSnapshot(PathResolverSnapshot snapshot, CancellationToken ct = default) { + var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(ct, _disposeToken.CancellationToken); + var linkedCt = linkedCts.Token; + return Task.Run(() => { + var userFiles = WorkspaceFiles(); + + // index library files if asked + CreateIndices(LibraryFiles(snapshot).Except(userFiles, FileSystemInfoComparer.Instance), _libraryCodeSymbolIndex, linkedCt); }, linkedCt).ContinueWith(_ => linkedCts.Dispose()); } + private void CreateIndices(IEnumerable files, ISymbolIndex symbolIndex, CancellationToken cancellationToken) { + foreach (var fileInfo in files) { + cancellationToken.ThrowIfCancellationRequested(); + if (ModulePath.IsPythonSourceFile(fileInfo.FullName)) { + symbolIndex.Parse(fileInfo.FullName); + } + } + } + private IEnumerable WorkspaceFiles() { if (string.IsNullOrEmpty(_workspaceRootPath)) { return Enumerable.Empty(); @@ -82,11 +106,20 @@ private IEnumerable WorkspaceFiles() { return _fileSystem.GetDirectoryInfo(_workspaceRootPath).EnumerateFileSystemInfos(_includeFiles, _excludeFiles); } + private IEnumerable LibraryFiles(PathResolverSnapshot snapshot) { + if (snapshot == null) { + return Enumerable.Empty(); + } + + var includeImplicit = !ModulePath.PythonVersionRequiresInitPyFiles(_version.ToVersion()); + return snapshot.GetAllImportableModuleFilePaths(includeImplicit).Select(p => new FileInfoProxy(new FileInfo(p))); + } + public void ProcessClosedFile(string path) { if (IsFileOnWorkspace(path)) { - _symbolIndex.Parse(path); + _userCodeSymbolIndex.Parse(path); } else { - _symbolIndex.Delete(path); + _userCodeSymbolIndex.Delete(path); } } @@ -99,28 +132,43 @@ private bool IsFileOnWorkspace(string path) { } public void ProcessNewFile(string path, IDocument doc) { - _symbolIndex.Add(path, doc); + _userCodeSymbolIndex.Add(path, doc); } public void ReIndexFile(string path, IDocument doc) { - _symbolIndex.ReIndex(path, doc); + _userCodeSymbolIndex.ReIndex(path, doc); } public void Dispose() { _disposables.TryDispose(); } - public Task> HierarchicalDocumentSymbolsAsync(string path, CancellationToken cancellationToken = default) { - return _symbolIndex.HierarchicalDocumentSymbolsAsync(path, cancellationToken); + public async Task> HierarchicalDocumentSymbolsAsync(string path, CancellationToken cancellationToken = default) { + var result = await _userCodeSymbolIndex.HierarchicalDocumentSymbolsAsync(path, cancellationToken); + if (result.Count > 0) { + return result; + } + + return await _libraryCodeSymbolIndex.HierarchicalDocumentSymbolsAsync(path, cancellationToken); } public Task> WorkspaceSymbolsAsync(string query, int maxLength, CancellationToken cancellationToken = default) { - return _symbolIndex.WorkspaceSymbolsAsync(query, maxLength, cancellationToken); + return WorkspaceSymbolsAsync(query, maxLength, includeLibraries: false, cancellationToken); + } + + public async Task> WorkspaceSymbolsAsync(string query, int maxLength, bool includeLibraries, CancellationToken cancellationToken = default) { + var userCodeResult = await _userCodeSymbolIndex.WorkspaceSymbolsAsync(query, maxLength, cancellationToken); + if (includeLibraries == false) { + return userCodeResult; + } + + var libraryCodeResult = await _libraryCodeSymbolIndex.WorkspaceSymbolsAsync(query, maxLength, cancellationToken); + return userCodeResult.Concat(libraryCodeResult).ToList(); } public void AddPendingDoc(IDocument doc) { _pendingDocs.TryAdd(doc, DateTime.Now); - _symbolIndex.MarkAsPending(doc.Uri.AbsolutePath); + _userCodeSymbolIndex.MarkAsPending(doc.Uri.AbsolutePath); } private void OnIdle(object sender, EventArgs _) { @@ -137,10 +185,21 @@ private void ReIndexPendingDocsAsync() { } private class UriDocumentComparer : IEqualityComparer { + public static readonly UriDocumentComparer Instance = new UriDocumentComparer(); + + private UriDocumentComparer() { } public bool Equals(IDocument x, IDocument y) => x.Uri.Equals(y.Uri); public int GetHashCode(IDocument obj) => obj.Uri.GetHashCode(); } + + private class FileSystemInfoComparer : IEqualityComparer { + public static readonly FileSystemInfoComparer Instance = new FileSystemInfoComparer(); + + private FileSystemInfoComparer() { } + public bool Equals(IFileSystemInfo x, IFileSystemInfo y) => x?.FullName == y?.FullName; + public int GetHashCode(IFileSystemInfo obj) => obj.FullName.GetHashCode(); + } } } diff --git a/src/LanguageServer/Impl/Indexing/MostRecentDocumentSymbols.cs b/src/LanguageServer/Impl/Indexing/MostRecentDocumentSymbols.cs index fe8b6969b..ae9593025 100644 --- a/src/LanguageServer/Impl/Indexing/MostRecentDocumentSymbols.cs +++ b/src/LanguageServer/Impl/Indexing/MostRecentDocumentSymbols.cs @@ -1,130 +1,140 @@ -using System; +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; using System.Collections.Generic; +using System.Collections.Immutable; using System.Diagnostics; using System.IO; using System.Threading; using System.Threading.Tasks; using Microsoft.Python.Analysis.Documents; using Microsoft.Python.Core; -using Microsoft.Python.Core.Diagnostics; using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.LanguageServer.Indexing { - class MostRecentDocumentSymbols : IMostRecentDocumentSymbols { - private readonly object _syncObj = new object(); + internal sealed class MostRecentDocumentSymbols : IMostRecentDocumentSymbols { private readonly IIndexParser _indexParser; private readonly string _path; + private readonly bool _library; - private CancellationTokenSource _fileCts = new CancellationTokenSource(); + // Only used to cancel all work when this object gets disposed. + private readonly CancellationTokenSource _cts = new CancellationTokenSource(); - private TaskCompletionSource> _fileTcs = new TaskCompletionSource>(); - private WorkQueueState state = WorkQueueState.WaitingForWork; + // Objects for the currently running task. + private readonly object _lock = new object(); + private TaskCompletionSource> _tcs = new TaskCompletionSource>(); + private CancellationTokenSource _workCts; - public MostRecentDocumentSymbols(string path, IIndexParser indexParser) { + public MostRecentDocumentSymbols(string path, IIndexParser indexParser, bool library) { _path = path; _indexParser = indexParser; + _library = library; } - public void Parse() { - WorkAndSetTcs(ParseAsync); + public Task> GetSymbolsAsync(CancellationToken ct = default) { + lock (_lock) { + return _tcs.Task.WaitAsync(ct); + } } - public void Index(IDocument doc) { - WorkAndSetTcs(ct => IndexAsync(doc, ct)); - } + public void Parse() => DoWork(ParseAsync); - public void WorkAndSetTcs(Func>> asyncWork) { - CancellationTokenSource currentCts; - TaskCompletionSource> currentTcs; - lock (_syncObj) { - switch (state) { - case WorkQueueState.Working: - CancelExistingWork(); - RenewTcs(); - break; - case WorkQueueState.WaitingForWork: - break; - case WorkQueueState.FinishedWork: - RenewTcs(); - break; - default: - throw new InvalidOperationException(); - } - state = WorkQueueState.Working; - currentCts = _fileCts; - currentTcs = _fileTcs; - } + public void Index(IDocument doc) => DoWork(ct => IndexAsync(doc, ct)); - DoWork(currentCts, asyncWork).SetCompletionResultTo(currentTcs); - } + private void DoWork(Func>> work) { + lock (_lock) { + // Invalidate any existing work. + MarkAsPending(); - private async Task> DoWork(CancellationTokenSource tcs, Func>> asyncWork) { - var token = tcs.Token; + // Create a new token for this specific work. + _workCts = CancellationTokenSource.CreateLinkedTokenSource(_cts.Token); - try { - return await asyncWork(token); - } finally { - lock (_syncObj) { - tcs.Dispose(); - if (!token.IsCancellationRequested) { - state = WorkQueueState.FinishedWork; + // Start the task and set the result to _tcs if the task doesn't get canceled. + UsingCts(_workCts, work).SetCompletionResultTo(_tcs, skipIfCanceled: true).DoNotWait(); + } + + // Because _workCts is created via CreateLinkedTokenSource, disposing of it sooner + // rather than later is important (as there are real unmanaged resources behind linked CTSs). + async Task> UsingCts( + CancellationTokenSource cts, + Func>> fn) { + using (cts) { + var result = await fn(cts.Token); + + lock (_lock) { + // Attempt to set _workCts to null if that's the current _workCts + // to avoid catching ObjectDisposedException all the time. + if (_workCts == cts) { + _workCts = null; + } } + + return result; } } } - public Task> GetSymbolsAsync(CancellationToken ct = default) { - TaskCompletionSource> currentTcs; - lock (_syncObj) { - currentTcs = _fileTcs; + public void MarkAsPending() { + lock (_lock) { + // Cancel the existing work, if any. + CancelWork(); + + // If the previous task was completed, then every task returned from GetSymbolsAsync + // will also be completed and it's too late to give them updated data. + // Create a new _tcs for future calls to GetSymbolsAsync to use. + // + // If the previous task wasn't completed, then we want to give the previous calls to + // GetSymbolsAsync the new result, so keep _tcs the same. + if (_tcs.Task.IsCompleted) { + _tcs = new TaskCompletionSource>(); + } } - return currentTcs.Task.ContinueWith(t => t.GetAwaiter().GetResult(), ct); } - public void MarkAsPending() { - lock (_syncObj) { - switch (state) { - case WorkQueueState.WaitingForWork: - break; - case WorkQueueState.Working: - CancelExistingWork(); - RenewTcs(); - break; - case WorkQueueState.FinishedWork: - RenewTcs(); - break; - default: - throw new InvalidOperationException(); + public void Dispose() { + lock (_lock) { + _tcs.TrySetCanceled(); + + try { + _workCts?.Dispose(); + } catch (ObjectDisposedException) { + // The task with this CTS completed and disposed already, ignore. } - state = WorkQueueState.WaitingForWork; + + _cts?.Dispose(); } } - public void Dispose() { - lock (_syncObj) { - switch (state) { - case WorkQueueState.Working: - CancelExistingWork(); - break; - case WorkQueueState.WaitingForWork: - CancelExistingWork(); - // Manually cancel tcs, in case any task is awaiting - _fileTcs.TrySetCanceled(); - break; - case WorkQueueState.FinishedWork: - break; - default: - throw new InvalidOperationException(); + private void CancelWork() { + if (_workCts != null) { + try { + _workCts.Cancel(); + } catch (ObjectDisposedException) { + // The task with this CTS completed and disposed already, ignore. } - state = WorkQueueState.FinishedWork; + _workCts = null; } } - private async Task> IndexAsync(IDocument doc, CancellationToken indexCt) { + private async Task> IndexAsync(IDocument doc, CancellationToken cancellationToken) { PythonAst ast = null; for (var i = 0; i < 5; i++) { - ast = await doc.GetAstAsync(indexCt); + cancellationToken.ThrowIfCancellationRequested(); + ast = await doc.GetAstAsync(cancellationToken); if (ast != null) { break; } @@ -132,42 +142,27 @@ private async Task> IndexAsync(IDocument doc, } if (ast == null) { - return Array.Empty(); + return ImmutableArray.Empty; } - indexCt.ThrowIfCancellationRequested(); - var walker = new SymbolIndexWalker(ast); + cancellationToken.ThrowIfCancellationRequested(); + var walker = new SymbolIndexWalker(ast, _library, cancellationToken); ast.Walk(walker); return walker.Symbols; } - private async Task> ParseAsync(CancellationToken parseCancellationToken) { + private async Task> ParseAsync(CancellationToken cancellationToken) { try { - var ast = await _indexParser.ParseAsync(_path, parseCancellationToken); - parseCancellationToken.ThrowIfCancellationRequested(); - var walker = new SymbolIndexWalker(ast); + var ast = await _indexParser.ParseAsync(_path, cancellationToken); + cancellationToken.ThrowIfCancellationRequested(); + var walker = new SymbolIndexWalker(ast, _library, cancellationToken); ast.Walk(walker); return walker.Symbols; } catch (Exception e) when (e is IOException || e is UnauthorizedAccessException) { Trace.TraceError(e.Message); } - return new List(); + return ImmutableArray.Empty; } - - private void RenewTcs() { - Check.InvalidOperation(Monitor.IsEntered(_syncObj)); - _fileCts = new CancellationTokenSource(); - _fileTcs = new TaskCompletionSource>(); - } - - private void CancelExistingWork() { - Check.InvalidOperation(Monitor.IsEntered(_syncObj)); - _fileCts.Cancel(); - } - - /* It's easier to think of it as a queue of work - * but it maintains only one item at a time in the queue */ - private enum WorkQueueState { WaitingForWork, Working, FinishedWork }; } } diff --git a/src/LanguageServer/Impl/Indexing/SymbolIndex.cs b/src/LanguageServer/Impl/Indexing/SymbolIndex.cs index 07628af36..3820c9d51 100644 --- a/src/LanguageServer/Impl/Indexing/SymbolIndex.cs +++ b/src/LanguageServer/Impl/Indexing/SymbolIndex.cs @@ -29,11 +29,13 @@ internal sealed class SymbolIndex : ISymbolIndex { private readonly DisposableBag _disposables = new DisposableBag(nameof(SymbolIndex)); private readonly ConcurrentDictionary _index; private readonly IIndexParser _indexParser; + private readonly bool _libraryMode; - public SymbolIndex(IFileSystem fileSystem, PythonLanguageVersion version) { - var comparer = PathEqualityComparer.Instance; - _index = new ConcurrentDictionary(comparer); + public SymbolIndex(IFileSystem fileSystem, PythonLanguageVersion version, bool libraryMode = false) { + _index = new ConcurrentDictionary(PathEqualityComparer.Instance); _indexParser = new IndexParser(fileSystem, version); + _libraryMode = libraryMode; + _disposables .Add(_indexParser) .Add(() => { @@ -89,28 +91,39 @@ public void MarkAsPending(string path) { } } - private IEnumerable WorkspaceSymbolsQuery(string path, string query, - IReadOnlyList symbols) { - var rootSymbols = DecorateWithParentsName(symbols, null); - var treeSymbols = rootSymbols.TraverseBreadthFirst((symAndPar) => { - var sym = symAndPar.symbol; - return DecorateWithParentsName((sym.Children ?? Enumerable.Empty()).ToList(), sym.Name); - }); - return treeSymbols.Where(sym => sym.symbol.Name.ContainsOrdinal(query, ignoreCase: true)) - .Select(sym => new FlatSymbol(sym.symbol.Name, sym.symbol.Kind, path, sym.symbol.SelectionRange, sym.parentName)); - } + private IEnumerable WorkspaceSymbolsQuery(string path, string query, IReadOnlyList symbols) { + var treeSymbols = DecorateWithParentsName(symbols, null) + .TraverseBreadthFirst(sym => DecorateWithParentsName(sym.symbol.Children ?? Enumerable.Empty(), sym.symbol.Name)); - private static IEnumerable<(HierarchicalSymbol symbol, string parentName)> DecorateWithParentsName( - IEnumerable symbols, string parentName) { - return symbols.Select((symbol) => (symbol, parentName)).ToList(); + // Prioritize exact matches, then substrings, then fuzzy matches. The caller is in charge of limiting this. + return treeSymbols.Where(sym => sym.symbol.Name.EqualsIgnoreCase(query)) + .Concat(treeSymbols.Where(sym => sym.symbol.Name.ContainsOrdinal(query, true))) + .Concat(treeSymbols.Where(sym => FuzzyMatch(query, sym.symbol.Name))) + .DistinctBy(sym => sym.symbol) // Deduplicate by reference; we're iterating over the exact same objects. + .Select(sym => new FlatSymbol(sym.symbol.Name, sym.symbol.Kind, path, sym.symbol.SelectionRange, sym.parentName, sym.symbol._existInAllVariable)); } - private IMostRecentDocumentSymbols MakeMostRecentDocSymbols(string path) { - return new MostRecentDocumentSymbols(path, _indexParser); - } + private IMostRecentDocumentSymbols MakeMostRecentDocSymbols(string path) => new MostRecentDocumentSymbols(path, _indexParser, _libraryMode); public void Dispose() { _disposables.TryDispose(); } + + private static IEnumerable<(HierarchicalSymbol symbol, string parentName)> DecorateWithParentsName(IEnumerable symbols, string parentName) + => symbols.Select((symbol) => (symbol, parentName)); + + private static bool FuzzyMatch(string pattern, string name) { + var patternPos = 0; + var namePos = 0; + + while (patternPos < pattern.Length && namePos < name.Length) { + if (char.ToLowerInvariant(pattern[patternPos]) == char.ToLowerInvariant(name[namePos])) { + patternPos++; + } + namePos++; + } + + return patternPos == pattern.Length; + } } } diff --git a/src/LanguageServer/Impl/Indexing/SymbolIndexWalker.cs b/src/LanguageServer/Impl/Indexing/SymbolIndexWalker.cs index 996c09762..c8ea2bfb4 100644 --- a/src/LanguageServer/Impl/Indexing/SymbolIndexWalker.cs +++ b/src/LanguageServer/Impl/Indexing/SymbolIndexWalker.cs @@ -16,6 +16,7 @@ using System.Collections.Generic; using System.Linq; using System.Text.RegularExpressions; +using System.Threading; using Microsoft.Python.Core; using Microsoft.Python.Parsing.Ast; @@ -25,17 +26,27 @@ internal class SymbolIndexWalker : PythonWalker { private static readonly Regex ConstantLike = new Regex(@"^[\p{Lu}\p{N}_]+$", RegexOptions.Compiled); private readonly PythonAst _ast; + private readonly bool _libraryMode; private readonly SymbolStack _stack = new SymbolStack(); + private readonly HashSet _namesInAllVariable; - public SymbolIndexWalker(PythonAst ast) { + public SymbolIndexWalker(PythonAst ast, bool libraryMode = false, CancellationToken cancellationToken = default) { _ast = ast; + _libraryMode = libraryMode; + + var collector = new AllVariableCollector(cancellationToken); + _ast.Walk(collector); + + _namesInAllVariable = collector.Names; } public IReadOnlyList Symbols => _stack.Root; public override bool Walk(ClassDefinition node) { _stack.Enter(SymbolKind.Class); - node.Body?.Walk(this); + + WalkIfNotLibraryMode(node.Body); + var children = _stack.Exit(); _stack.AddSymbol(new HierarchicalSymbol( @@ -44,7 +55,8 @@ public override bool Walk(ClassDefinition node) { node.GetSpan(_ast), node.NameExpression.GetSpan(_ast), children, - FunctionKind.Class + FunctionKind.Class, + ExistInAllVariable(node.Name) )); return false; @@ -55,75 +67,66 @@ public override bool Walk(FunctionDefinition node) { foreach (var p in node.Parameters) { AddVarSymbol(p.NameExpression); } - node.Body?.Walk(this); + + // don't bother to walk down locals for libraries + // we don't care those for libraries + WalkIfNotLibraryMode(node.Body); + var children = _stack.Exit(); + SymbolKind symbolKind; + string functionKind; + GetKinds(node, out symbolKind, out functionKind); + var span = node.GetSpan(_ast); var ds = new HierarchicalSymbol( node.Name, - SymbolKind.Function, + symbolKind, span, node.IsLambda ? span : node.NameExpression.GetSpan(_ast), children, - FunctionKind.Function + functionKind, + ExistInAllVariable(node.Name) ); + _stack.AddSymbol(ds); + return false; + } + + private void GetKinds(FunctionDefinition node, out SymbolKind symbolKind, out string functionKind) { + symbolKind = SymbolKind.Function; + functionKind = FunctionKind.Function; + if (_stack.Parent == SymbolKind.Class) { - switch (ds.Name) { + switch (node.Name) { case "__init__": - ds.Kind = SymbolKind.Constructor; + symbolKind = SymbolKind.Constructor; break; case var name when DoubleUnderscore.IsMatch(name): - ds.Kind = SymbolKind.Operator; + symbolKind = SymbolKind.Operator; break; default: - ds.Kind = SymbolKind.Method; + symbolKind = SymbolKind.Method; if (node.Decorators != null) { foreach (var dec in node.Decorators.Decorators) { var maybeKind = DecoratorExpressionToKind(dec); if (maybeKind.HasValue) { - ds.Kind = maybeKind.Value.kind; - ds._functionKind = maybeKind.Value.functionKind; + symbolKind = maybeKind.Value.kind; + functionKind = maybeKind.Value.functionKind; break; } } } - break; } } - - _stack.AddSymbol(ds); - - return false; - } - - public override bool Walk(ImportStatement node) { - foreach (var (nameNode, nameString) in node.Names.Zip(node.AsNames, (name, asName) => asName != null ? (asName, asName.Name) : ((Node)name, name.MakeString()))) { - var span = nameNode.GetSpan(_ast); - _stack.AddSymbol(new HierarchicalSymbol(nameString, SymbolKind.Module, span)); - } - - return false; - } - - public override bool Walk(FromImportStatement node) { - if (node.IsFromFuture) { - return false; - } - - foreach (var name in node.Names.Zip(node.AsNames, (name, asName) => asName ?? name)) { - var span = name.GetSpan(_ast); - _stack.AddSymbol(new HierarchicalSymbol(name.Name, SymbolKind.Module, span)); - } - - return false; } public override bool Walk(AssignmentStatement node) { - node.Right?.Walk(this); + WalkIfNotLibraryMode(node.Right); + foreach (var exp in node.Left) { AddVarSymbolRecursive(exp); } @@ -132,13 +135,15 @@ public override bool Walk(AssignmentStatement node) { } public override bool Walk(NamedExpression node) { - node.Value?.Walk(this); + WalkIfNotLibraryMode(node.Value); + AddVarSymbolRecursive(node.Target); return false; } public override bool Walk(AugmentedAssignStatement node) { - node.Right?.Walk(this); + WalkIfNotLibraryMode(node.Right); + AddVarSymbol(node.Left as NameExpression); return false; } @@ -174,11 +179,19 @@ public override bool Walk(ForStatement node) { } public override bool Walk(ComprehensionFor node) { + if (_libraryMode) { + return false; + } + AddVarSymbolRecursive(node.Left); return base.Walk(node); } public override bool Walk(ListComprehension node) { + if (_libraryMode) { + return false; + } + _stack.Enter(SymbolKind.None); return base.Walk(node); } @@ -186,6 +199,10 @@ public override bool Walk(ListComprehension node) { public override void PostWalk(ListComprehension node) => ExitComprehension(node); public override bool Walk(DictionaryComprehension node) { + if (_libraryMode) { + return false; + } + _stack.Enter(SymbolKind.None); return base.Walk(node); } @@ -193,6 +210,10 @@ public override bool Walk(DictionaryComprehension node) { public override void PostWalk(DictionaryComprehension node) => ExitComprehension(node); public override bool Walk(SetComprehension node) { + if (_libraryMode) { + return false; + } + _stack.Enter(SymbolKind.None); return base.Walk(node); } @@ -200,6 +221,10 @@ public override bool Walk(SetComprehension node) { public override void PostWalk(SetComprehension node) => ExitComprehension(node); public override bool Walk(GeneratorExpression node) { + if (_libraryMode) { + return false; + } + _stack.Enter(SymbolKind.None); return base.Walk(node); } @@ -207,6 +232,10 @@ public override bool Walk(GeneratorExpression node) { public override void PostWalk(GeneratorExpression node) => ExitComprehension(node); private void ExitComprehension(Comprehension node) { + if (_libraryMode) { + return; + } + var children = _stack.Exit(); var span = node.GetSpan(_ast); @@ -237,7 +266,7 @@ private void AddVarSymbol(NameExpression node) { var span = node.GetSpan(_ast); - _stack.AddSymbol(new HierarchicalSymbol(node.Name, kind, span)); + _stack.AddSymbol(new HierarchicalSymbol(node.Name, kind, span, existInAllVariable: ExistInAllVariable(node.Name))); } private void AddVarSymbolRecursive(Expression node) { @@ -281,6 +310,14 @@ private void AddVarSymbolRecursive(Expression node) { return null; } + private void WalkIfNotLibraryMode(Node node) { + if (_libraryMode) { + return; + } + + node?.Walk(this); + } + private bool NameIsProperty(string name) => name == "property" || name == "abstractproperty" @@ -311,16 +348,21 @@ private void WalkAndDeclareAll(IEnumerable nodes) { } } + private bool ExistInAllVariable(string name) { + return _namesInAllVariable.Contains(name); + } + private class SymbolStack { private readonly Stack<(SymbolKind? kind, List symbols)> _symbols; private readonly Stack> _declared = new Stack>(new[] { new HashSet() }); + private readonly List _root = new List(); - public List Root { get; } = new List(); + public IReadOnlyList Root => _root; public SymbolKind? Parent => _symbols.Peek().kind; public SymbolStack() { - _symbols = new Stack<(SymbolKind?, List)>(new (SymbolKind?, List)[] { (null, Root) }); + _symbols = new Stack<(SymbolKind?, List)>(new (SymbolKind?, List)[] { (null, _root) }); } public void Enter(SymbolKind parent) { @@ -339,6 +381,10 @@ public List Exit() { public void ExitDeclaredAndMerge() => _declared.Peek().UnionWith(_declared.Pop()); public void AddSymbol(HierarchicalSymbol sym) { + if (string.IsNullOrWhiteSpace(sym.Name)) { + return; + } + if (sym.Kind == SymbolKind.Variable && _declared.Peek().Contains(sym.Name)) { return; } diff --git a/src/LanguageServer/Impl/Indexing/Symbols.cs b/src/LanguageServer/Impl/Indexing/Symbols.cs index 0cdac218d..eecb55725 100644 --- a/src/LanguageServer/Impl/Indexing/Symbols.cs +++ b/src/LanguageServer/Impl/Indexing/Symbols.cs @@ -14,6 +14,7 @@ // permissions and limitations under the License. using System.Collections.Generic; +using System.Diagnostics; using Microsoft.Python.Core.Text; namespace Microsoft.Python.LanguageServer.Indexing { @@ -58,16 +59,18 @@ internal class FunctionKind { } // Analagous to LSP's DocumentSymbol. - internal class HierarchicalSymbol { - public string Name; - public string Detail; - public SymbolKind Kind; - public bool? Deprecated; - public SourceSpan Range; - public SourceSpan SelectionRange; - public IList Children; + [DebuggerDisplay("{Name}, {Kind}")] + internal sealed class HierarchicalSymbol { + public readonly string Name; + public readonly string Detail; + public readonly SymbolKind Kind; + public readonly bool? Deprecated; + public readonly SourceSpan Range; + public readonly SourceSpan SelectionRange; + public readonly IList Children; - public string _functionKind; + public readonly string _functionKind; + public readonly bool _existInAllVariable; public HierarchicalSymbol( string name, @@ -75,7 +78,8 @@ public HierarchicalSymbol( SourceSpan range, SourceSpan? selectionRange = null, IList children = null, - string functionKind = FunctionKind.None + string functionKind = FunctionKind.None, + bool existInAllVariable = false ) { Name = name; Kind = kind; @@ -83,30 +87,36 @@ public HierarchicalSymbol( SelectionRange = selectionRange ?? range; Children = children; _functionKind = functionKind; + _existInAllVariable = existInAllVariable; } } // Analagous to LSP's SymbolInformation. - internal class FlatSymbol { - public string Name; - public SymbolKind Kind; - public bool? Deprecated; - public string DocumentPath; - public SourceSpan Range; - public string ContainerName; + [DebuggerDisplay("{ContainerName}:{Name}, {Kind}")] + internal sealed class FlatSymbol { + public readonly string Name; + public readonly SymbolKind Kind; + public readonly bool? Deprecated; + public readonly string DocumentPath; + public readonly SourceSpan Range; + public readonly string ContainerName; + + public readonly bool _existInAllVariable; public FlatSymbol( string name, SymbolKind kind, string documentPath, SourceSpan range, - string containerName = null + string containerName = null, + bool existInAllVariable = false ) { Name = name; Kind = kind; DocumentPath = documentPath; Range = range; ContainerName = containerName; + _existInAllVariable = existInAllVariable; } } } diff --git a/src/LanguageServer/Impl/LanguageServer.Configuration.cs b/src/LanguageServer/Impl/LanguageServer.Configuration.cs index 6a21b2f1c..0a286f30f 100644 --- a/src/LanguageServer/Impl/LanguageServer.Configuration.cs +++ b/src/LanguageServer/Impl/LanguageServer.Configuration.cs @@ -29,6 +29,7 @@ using Microsoft.Python.Core.Collections; using Microsoft.Python.Core.IO; using Microsoft.Python.Core.OS; +using Microsoft.Python.LanguageServer.CodeActions; using Microsoft.Python.LanguageServer.Protocol; using Microsoft.Python.LanguageServer.SearchPaths; using Newtonsoft.Json.Linq; @@ -40,6 +41,8 @@ public sealed partial class LanguageServer { [JsonRpcMethod("workspace/didChangeConfiguration")] public async Task DidChangeConfiguration(JToken token, CancellationToken cancellationToken) { using (await _prioritizer.ConfigurationPriorityAsync(cancellationToken)) { + Debug.Assert(_initialized); + var settings = new LanguageServerSettings(); // https://github.com/microsoft/python-language-server/issues/915 @@ -63,17 +66,56 @@ public async Task DidChangeConfiguration(JToken token, CancellationToken cancell var userConfiguredPaths = GetUserConfiguredPaths(pythonSection); HandleUserConfiguredPathsChanges(userConfiguredPaths); - HandlePathWatchChanges(GetSetting(analysis, "watchSearchPaths", true)); + HandlePathWatchChanges(GetSetting(analysis, "watchSearchPaths", false)); HandleDiagnosticsChanges(pythonSection, settings); + HandleCodeActionsChanges(pythonSection); _server.DidChangeConfiguration(new DidChangeConfigurationParams { settings = settings }, cancellationToken); } } + private void HandleCodeActionsChanges(JToken pythonSection) { + var refactoring = new Dictionary(); + var quickFix = new Dictionary(); + + var refactoringToken = pythonSection["refactoring"]; + var quickFixToken = pythonSection["quickfix"]; + + // +1 is for last "." after prefix + AppendToMap(refactoringToken, refactoringToken?.Path.Length + 1 ?? 0, refactoring); + AppendToMap(quickFixToken, quickFixToken?.Path.Length + 1 ?? 0, quickFix); + + var codeActionSettings = new CodeActionSettings(refactoring, quickFix); + _server.HandleCodeActionsChange(codeActionSettings); + + void AppendToMap(JToken setting, int prefixLength, Dictionary map) { + if (setting == null || !setting.HasValues) { + return; + } + + foreach (var child in setting) { + if (child is JValue value) { + // there shouldn't be duplicates and prefix must exist. + var path = child.Path; + if (path.Length <= prefixLength) { + // nothing to add + continue; + } + + // get rid of common "settings.python..." prefix + map[path.Substring(prefixLength)] = value.Value; + continue; + } + + AppendToMap(child, prefixLength, map); + } + } + } + private void HandleDiagnosticsChanges(JToken pythonSection, LanguageServerSettings settings) { var analysis = pythonSection["analysis"]; - settings.diagnosticPublishDelay = GetSetting(analysis, "diagnosticPublishDelay", 1000); + settings.diagnosticPublishDelay = GetSetting(analysis, "diagnosticPublishDelay", 200); var ds = _services.GetService(); ds.PublishingDelay = settings.diagnosticPublishDelay; @@ -88,7 +130,6 @@ private void HandleDiagnosticsChanges(JToken pythonSection, LanguageServerSettin var memory = analysis["memory"]; var optionsProvider = _services.GetService(); - optionsProvider.Options.KeepLibraryLocalVariables = GetSetting(memory, "keepLibraryLocalVariables", false); optionsProvider.Options.KeepLibraryAst = GetSetting(memory, "keepLibraryAst", false); optionsProvider.Options.AnalysisCachingLevel = GetAnalysisCachingLevel(analysis); @@ -180,12 +221,24 @@ private ImmutableArray GetUserConfiguredPaths(JToken pythonSection) { return ImmutableArray.Empty; } + private const string DefaultCachingLevel = "None"; + private AnalysisCachingLevel GetAnalysisCachingLevel(JToken analysisKey) { - var s = GetSetting(analysisKey, "cachingLevel", "None"); - if (s.EqualsIgnoreCase("System")) { - return AnalysisCachingLevel.System; - } - return s.EqualsIgnoreCase("Library") ? AnalysisCachingLevel.Library : AnalysisCachingLevel.None; + // TODO: Remove this one caching is working at any level again. + // https://github.com/microsoft/python-language-server/issues/1758 + return AnalysisCachingLevel.None; + + // var s = GetSetting(analysisKey, "cachingLevel", DefaultCachingLevel); + // + // if (string.IsNullOrWhiteSpace(s) || s.EqualsIgnoreCase("Default")) { + // s = DefaultCachingLevel; + // } + // + // if (s.EqualsIgnoreCase("System")) { + // return AnalysisCachingLevel.System; + // } + // + // return s.EqualsIgnoreCase("Library") ? AnalysisCachingLevel.Library : AnalysisCachingLevel.None; } } } diff --git a/src/LanguageServer/Impl/LanguageServer.Lifetime.cs b/src/LanguageServer/Impl/LanguageServer.Lifetime.cs index f09b44db4..811d7bf22 100644 --- a/src/LanguageServer/Impl/LanguageServer.Lifetime.cs +++ b/src/LanguageServer/Impl/LanguageServer.Lifetime.cs @@ -14,11 +14,12 @@ // permissions and limitations under the License. using System; -using System.Collections.Generic; using System.Diagnostics; using System.Threading; using System.Threading.Tasks; +using Microsoft.Python.Analysis.Caching; using Microsoft.Python.Core; +using Microsoft.Python.LanguageServer.Optimization; using Microsoft.Python.LanguageServer.Protocol; using Newtonsoft.Json.Linq; using StreamJsonRpc; @@ -27,6 +28,7 @@ namespace Microsoft.Python.LanguageServer.Implementation { public partial class LanguageServer { private InitializeParams _initParams; private bool _shutdown; + private bool _initialized; private Task _initializedPriorityTask; @@ -34,21 +36,29 @@ public partial class LanguageServer { public async Task Initialize(JToken token, CancellationToken cancellationToken) { _initParams = token.ToObject(); MonitorParentProcess(_initParams); + RegisterServices(_initParams); + using (await _prioritizer.InitializePriorityAsync(cancellationToken)) { + Debug.Assert(!_initialized); // Force the next handled request to be "initialized", where the work actually happens. - _initializedPriorityTask = _prioritizer.InitializePriorityAsync(default); - return await _server.InitializeAsync(_initParams, cancellationToken); + _initializedPriorityTask = _prioritizer.InitializedPriorityAsync(); + var result = await _server.InitializeAsync(_initParams, cancellationToken); + return result; } } [JsonRpcMethod("initialized")] public async Task Initialized(JToken token, CancellationToken cancellationToken) { + _services.GetService()?.Profile("Initialized"); + using (await _initializedPriorityTask) { + Debug.Assert(!_initialized); var pythonSection = await GetPythonConfigurationAsync(cancellationToken, 200); var userConfiguredPaths = GetUserConfiguredPaths(pythonSection); await _server.InitializedAsync(ToObject(token), cancellationToken, userConfiguredPaths); await _rpc.NotifyAsync("python/languageServerStarted"); + _initialized = true; } } @@ -64,7 +74,10 @@ private async Task GetPythonConfigurationAsync(CancellationToken cancell } var args = new ConfigurationParams { items = new[] { - new ConfigurationItem { section = "python" } + new ConfigurationItem { + scopeUri = _initParams.rootUri, + section = "python" + } } }; var configs = await _rpc.InvokeWithParameterObjectAsync("workspace/configuration", args, cancellationToken); @@ -103,7 +116,7 @@ private void MonitorParentProcess(InitializeParams p) { Debug.Assert(parentProcess != null, "Parent process does not exist"); if (parentProcess != null) { - parentProcess.Exited += (s, e) => _sessionTokenSource.Cancel(); + parentProcess.Exited += (s, e) => TerminateProcess(); } } @@ -112,11 +125,23 @@ private void MonitorParentProcess(InitializeParams p) { while (!_sessionTokenSource.IsCancellationRequested) { await Task.Delay(2000); if (parentProcess.HasExited) { - _sessionTokenSource.Cancel(); + TerminateProcess(); } } }).DoNotWait(); } } + + private void TerminateProcess() { + _sessionTokenSource?.Cancel(); + Environment.Exit(0); + } + + private void RegisterServices(InitializeParams initParams) { + // we need to register cache service first. + // optimization service consumes the cache info. + CacheService.Register(_services, initParams?.initializationOptions?.cacheFolderPath); + _services.AddService(new ProfileOptimizationService(_services)); + } } } diff --git a/src/LanguageServer/Impl/LanguageServer.cs b/src/LanguageServer/Impl/LanguageServer.cs index ca4d1e962..3df9d7b1d 100644 --- a/src/LanguageServer/Impl/LanguageServer.cs +++ b/src/LanguageServer/Impl/LanguageServer.cs @@ -15,7 +15,6 @@ // permissions and limitations under the License. using System; -using System.Collections.Generic; using System.Diagnostics; using System.Threading; using System.Threading.Tasks; @@ -32,7 +31,6 @@ using Newtonsoft.Json; using Newtonsoft.Json.Linq; using StreamJsonRpc; -using Range = Microsoft.Python.Core.Text.Range; namespace Microsoft.Python.LanguageServer.Implementation { /// @@ -47,7 +45,7 @@ public sealed partial class LanguageServer : IDisposable { private readonly CancellationTokenSource _shutdownCts = new CancellationTokenSource(); private readonly AnalysisOptionsProvider _optionsProvider = new AnalysisOptionsProvider(); - private IServiceContainer _services; + private IServiceManager _services; private Server _server; private ILogger _logger; private ITelemetryService _telemetry; @@ -93,6 +91,7 @@ private void OnApplyWorkspaceEdit(object sender, ApplyWorkspaceEditEventArgs e) [JsonRpcMethod("workspace/didChangeWatchedFiles")] public async Task DidChangeWatchedFiles(JToken token, CancellationToken cancellationToken) { using (await _prioritizer.DocumentChangePriorityAsync(cancellationToken)) { + Debug.Assert(_initialized); _server.DidChangeWatchedFiles(ToObject(token)); } } @@ -101,6 +100,7 @@ public async Task DidChangeWatchedFiles(JToken token, CancellationToken cancella public async Task WorkspaceSymbols(JToken token, CancellationToken cancellationToken) { using (var timer = _requestTimer.Time("workspace/symbol")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); var result = await _server.WorkspaceSymbols(ToObject(token), cancellationToken); timer.AddMeasure("count", result?.Length ?? 0); return result; @@ -120,6 +120,7 @@ public async Task WorkspaceSymbols(JToken token, Cancellati public async Task DidOpenTextDocument(JToken token, CancellationToken cancellationToken) { _idleTimeTracker?.NotifyUserActivity(); using (await _prioritizer.DocumentChangePriorityAsync(cancellationToken)) { + Debug.Assert(_initialized); _server.DidOpenTextDocument(ToObject(token)); } } @@ -128,6 +129,7 @@ public async Task DidOpenTextDocument(JToken token, CancellationToken cancellati public async Task DidChangeTextDocument(JToken token, CancellationToken cancellationToken) { _idleTimeTracker?.NotifyUserActivity(); using (await _prioritizer.DocumentChangePriorityAsync(cancellationToken)) { + Debug.Assert(_initialized); var @params = ToObject(token); _server.DidChangeTextDocument(@params); } @@ -146,6 +148,7 @@ public void WillSaveTextDocument(JToken token) { } public async Task DidCloseTextDocument(JToken token, CancellationToken cancellationToken) { _idleTimeTracker?.NotifyUserActivity(); using (await _prioritizer.DocumentChangePriorityAsync(cancellationToken)) { + Debug.Assert(_initialized); _server.DidCloseTextDocument(ToObject(token)); } } @@ -156,6 +159,7 @@ public async Task DidCloseTextDocument(JToken token, CancellationToken cancellat public async Task Completion(JToken token, CancellationToken cancellationToken) { using (var timer = _requestTimer.Time("textDocument/completion")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); var result = await _server.Completion(ToObject(token), GetToken(cancellationToken)); timer.AddMeasure("count", result?.items?.Length ?? 0); return result; @@ -166,6 +170,7 @@ public async Task Completion(JToken token, CancellationToken can public async Task Hover(JToken token, CancellationToken cancellationToken) { using (_requestTimer.Time("textDocument/hover")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.Hover(ToObject(token), GetToken(cancellationToken)); } } @@ -174,6 +179,7 @@ public async Task Hover(JToken token, CancellationToken cancellationToken public async Task SignatureHelp(JToken token, CancellationToken cancellationToken) { using (_requestTimer.Time("textDocument/signatureHelp")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.SignatureHelp(ToObject(token), GetToken(cancellationToken)); } } @@ -182,6 +188,7 @@ public async Task SignatureHelp(JToken token, CancellationToken c public async Task GotoDefinition(JToken token, CancellationToken cancellationToken) { using (_requestTimer.Time("textDocument/definition")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.GotoDefinition(ToObject(token), GetToken(cancellationToken)); } } @@ -190,6 +197,7 @@ public async Task GotoDefinition(JToken token, CancellationToken ca public async Task GotoDeclaration(JToken token, CancellationToken cancellationToken) { using (_requestTimer.Time("textDocument/declaration")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.GotoDeclaration(ToObject(token), GetToken(cancellationToken)); } } @@ -198,30 +206,39 @@ public async Task GotoDeclaration(JToken token, CancellationToken canc public async Task FindReferences(JToken token, CancellationToken cancellationToken) { using (_requestTimer.Time("textDocument/references")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.FindReferences(ToObject(token), GetToken(cancellationToken)); } } - //[JsonRpcMethod("textDocument/documentHighlight")] - //public async Task DocumentHighlight(JToken token, CancellationToken cancellationToken) { - // await _prioritizer.DefaultPriorityAsync(cancellationToken); - // return await _server.DocumentHighlight(ToObject(token), cancellationToken); - //} + [JsonRpcMethod("textDocument/documentHighlight")] + public async Task DocumentHighlight(JToken token, CancellationToken cancellationToken) { + using (_requestTimer.Time("textDocument/documentHighlight")) { + await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); + return await _server.DocumentHighlight(ToObject(token), cancellationToken); + } + } [JsonRpcMethod("textDocument/documentSymbol")] public async Task DocumentSymbol(JToken token, CancellationToken cancellationToken) { using (_requestTimer.Time("textDocument/documentSymbol")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); // This call is also used by VSC document outline and it needs correct information return await _server.HierarchicalDocumentSymbol(ToObject(token), GetToken(cancellationToken)); } } - //[JsonRpcMethod("textDocument/codeAction")] - //public async Task CodeAction(JToken token, CancellationToken cancellationToken) { - // await _prioritizer.DefaultPriorityAsync(cancellationToken); - // return await _server.CodeAction(ToObject(token), cancellationToken); - //} + [JsonRpcMethod("textDocument/codeAction")] + public async Task CodeAction(JToken token, CancellationToken cancellationToken) { + using var timer = _requestTimer.Time("textDocument/codeAction"); + await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); + var actions = await _server.CodeAction(ToObject(token), cancellationToken); + timer.AddMeasure("count", actions?.Length ?? 0); + return actions; + } //[JsonRpcMethod("textDocument/codeLens")] //public async Task CodeLens(JToken token, CancellationToken cancellationToken) { @@ -258,6 +275,7 @@ public async Task DocumentSymbol(JToken token, CancellationTok [JsonRpcMethod("textDocument/onTypeFormatting")] public async Task DocumentOnTypeFormatting(JToken token, CancellationToken cancellationToken) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.DocumentOnTypeFormatting(ToObject(token), GetToken(cancellationToken)); } @@ -265,6 +283,7 @@ public async Task DocumentOnTypeFormatting(JToken token, Cancellatio public async Task Rename(JToken token, CancellationToken cancellationToken) { using (var timer = _requestTimer.Time("textDocument/rename")) { await _prioritizer.DefaultPriorityAsync(cancellationToken); + Debug.Assert(_initialized); return await _server.Rename(ToObject(token), GetToken(cancellationToken)); } } @@ -287,6 +306,16 @@ public async Task ExtensionCommand(JToken token, CancellationToken cancellationT } #endregion + #region Custom + [JsonRpcMethod("python/clearAnalysisCache")] + public async Task ClearAnalysisCache(CancellationToken cancellationToken) { + using (_requestTimer.Time("python/clearAnalysisCache")) + using (await _prioritizer.ConfigurationPriorityAsync(cancellationToken)) { + _server.ClearAnalysisCache(); + } + } + #endregion + private T ToObject(JToken token) => token.ToObject(_jsonSerializer); private T GetSetting(JToken section, string settingName, T defaultValue) { @@ -317,14 +346,19 @@ private static CancellationToken GetToken(CancellationToken original) => Debugger.IsAttached ? CancellationToken.None : original; private class Prioritizer : IDisposable { - private const int InitializePriority = 0; - private const int ConfigurationPriority = 1; - private const int DocumentChangePriority = 2; - private const int DefaultPriority = 3; + private enum Priority { + Initialize, + Initialized, + Configuration, + DocumentChange, + Default, + Length, // Length of the enum, not a real priority. + } + private readonly PriorityProducerConsumer _ppc; public Prioritizer() { - _ppc = new PriorityProducerConsumer(4); + _ppc = new PriorityProducerConsumer(maxPriority: (int)Priority.Length); Task.Run(ConsumerLoop).DoNotWait(); } @@ -345,21 +379,24 @@ private async Task ConsumerLoop() { } } - public Task InitializePriorityAsync(CancellationToken cancellationToken = default(CancellationToken)) - => Enqueue(InitializePriority, true, cancellationToken); + public Task InitializePriorityAsync(CancellationToken cancellationToken = default) + => Enqueue(Priority.Initialize, isAwaitable: true, cancellationToken); + + public Task InitializedPriorityAsync(CancellationToken cancellationToken = default) + => Enqueue(Priority.Initialized, isAwaitable: true, cancellationToken); - public Task ConfigurationPriorityAsync(CancellationToken cancellationToken = default(CancellationToken)) - => Enqueue(ConfigurationPriority, true, cancellationToken); + public Task ConfigurationPriorityAsync(CancellationToken cancellationToken = default) + => Enqueue(Priority.Configuration, isAwaitable: true, cancellationToken); - public Task DocumentChangePriorityAsync(CancellationToken cancellationToken = default(CancellationToken)) - => Enqueue(DocumentChangePriority, true, cancellationToken); + public Task DocumentChangePriorityAsync(CancellationToken cancellationToken = default) + => Enqueue(Priority.DocumentChange, isAwaitable: true, cancellationToken); - public Task DefaultPriorityAsync(CancellationToken cancellationToken = default(CancellationToken)) - => Enqueue(DefaultPriority, false, cancellationToken); + public Task DefaultPriorityAsync(CancellationToken cancellationToken = default) + => Enqueue(Priority.Default, isAwaitable: false, cancellationToken); - private Task Enqueue(int priority, bool isAwaitable, CancellationToken cancellationToken = default(CancellationToken)) { + private Task Enqueue(Priority priority, bool isAwaitable, CancellationToken cancellationToken) { var item = new QueueItem(isAwaitable, cancellationToken); - _ppc.Produce(item, priority); + _ppc.Produce(item, (int)priority); return item.Task; } diff --git a/src/LanguageServer/Impl/LanguageServerSettings.cs b/src/LanguageServer/Impl/LanguageServerSettings.cs index c796205f0..5d4c65368 100644 --- a/src/LanguageServer/Impl/LanguageServerSettings.cs +++ b/src/LanguageServer/Impl/LanguageServerSettings.cs @@ -16,7 +16,7 @@ namespace Microsoft.Python.LanguageServer.Implementation { public sealed class LanguageServerSettings: ServerSettings { - public int diagnosticPublishDelay = 1000; + public int diagnosticPublishDelay = 200; public int symbolsHierarchyDepthLimit = 10; public int symbolsHierarchyMaxSymbols = 1000; } diff --git a/src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj b/src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj index ca455737f..ed8430b17 100644 --- a/src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj +++ b/src/LanguageServer/Impl/Microsoft.Python.LanguageServer.csproj @@ -1,6 +1,6 @@  - netcoreapp3.0 + netcoreapp3.1 Microsoft.Python.LanguageServer Microsoft.Python.LanguageServer @@ -11,7 +11,7 @@ portable true 1701;1702;$(NoWarn) - 7.2 + true ..\..\PLS.ruleset @@ -29,21 +29,12 @@ all runtime; build; native; contentfiles; analyzers - - - - - - - $(AnalysisReference)/Microsoft.Python.Analysis.Engine.dll - - - PreserveNewest - + + + - diff --git a/src/LanguageServer/Impl/Optimization/IProfileOptimizationService.cs b/src/LanguageServer/Impl/Optimization/IProfileOptimizationService.cs new file mode 100644 index 000000000..1610fc532 --- /dev/null +++ b/src/LanguageServer/Impl/Optimization/IProfileOptimizationService.cs @@ -0,0 +1,28 @@ +// Python Tools for Visual Studio +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Runtime; + +namespace Microsoft.Python.LanguageServer.Optimization { + /// + /// Service to enable profile based optimization + /// + /// See for more detail + /// + internal interface IProfileOptimizationService { + void Profile(string name); + } +} diff --git a/src/LanguageServer/Impl/Optimization/ProfileOptimizationService.cs b/src/LanguageServer/Impl/Optimization/ProfileOptimizationService.cs new file mode 100644 index 000000000..ecd6c69f9 --- /dev/null +++ b/src/LanguageServer/Impl/Optimization/ProfileOptimizationService.cs @@ -0,0 +1,47 @@ +// Python Tools for Visual Studio +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.IO; +using System.Runtime; +using Microsoft.Python.Analysis.Caching; +using Microsoft.Python.Core; +using Microsoft.Python.Core.IO; + +namespace Microsoft.Python.LanguageServer.Optimization { + internal sealed class ProfileOptimizationService : IProfileOptimizationService { + public ProfileOptimizationService(IServiceContainer services) { + var fileSystem = services.GetService(); + var cacheService = services.GetService(); + if (fileSystem == null || cacheService == null) { + return; + } + + try { + // create directory for profile optimization + var path = Path.Combine(cacheService.CacheFolder, "Profiles"); + fileSystem.CreateDirectory(path); + + ProfileOptimization.SetProfileRoot(path); + } catch { + // ignore any issue with profiling + } + } + + public void Profile(string name) { + ProfileOptimization.StartProfile(name); + } + } +} diff --git a/src/LanguageServer/Impl/Program.cs b/src/LanguageServer/Impl/Program.cs index 024f9af18..25528cdb5 100644 --- a/src/LanguageServer/Impl/Program.cs +++ b/src/LanguageServer/Impl/Program.cs @@ -31,6 +31,7 @@ namespace Microsoft.Python.LanguageServer.Server { internal static class Program { public static void Main(string[] args) { CheckDebugMode(); + using (CoreShell.Create()) { var services = CoreShell.Current.ServiceManager; diff --git a/src/LanguageServer/Impl/Protocol/Classes.cs b/src/LanguageServer/Impl/Protocol/Classes.cs index bdb39e38a..88842f6e6 100644 --- a/src/LanguageServer/Impl/Protocol/Classes.cs +++ b/src/LanguageServer/Impl/Protocol/Classes.cs @@ -363,7 +363,32 @@ public sealed class DefinitionCapabilities { public bool dynamicRegistration; } public DefinitionCapabilities definition; [Serializable] - public sealed class CodeActionCapabilities { public bool dynamicRegistration; } + public sealed class CodeActionCapabilities { + public bool dynamicRegistration; + // + // The client support code action literals as a valid + // response of the `textDocument/codeAction` request. + // + // Since 3.8.0 + // + public class CodeActionLiteralSupport { + // + // The code action kind is support with the following value + // set. + // + public class CodeActionKind { + // + // The code action kind values the client supports. When this + // property exists the client also guarantees that it will + // handle values outside its set gracefully and falls back + // to a default value when unknown. + // + public string[] valueSet; + } + public CodeActionKind codeActionKind; + } + public CodeActionLiteralSupport codeActionLiteralSupport; + } public CodeActionCapabilities codeAction; [Serializable] @@ -412,6 +437,17 @@ public sealed class CodeLensOptions { public bool resolveProvider; } + [Serializable] + public sealed class CodeActionOptions { + // + // CodeActionKinds that this server may return. + // + // The list of kinds may be generic, such as `CodeActionKind.Refactor`, or the server + // may list out every specific kind they provide. + // + public string[] codeActionKinds; + } + [Serializable] public sealed class DocumentOnTypeFormattingOptions { public string firstTriggerCharacter; @@ -456,7 +492,7 @@ public sealed class ServerCapabilities { public bool documentHighlightProvider; public bool documentSymbolProvider; public bool workspaceSymbolProvider; - public bool codeActionProvider; + public CodeActionOptions codeActionProvider; public CodeLensOptions codeLensProvider; public bool documentFormattingProvider; public bool documentRangeFormattingProvider; @@ -741,4 +777,35 @@ public sealed class PublishDiagnosticsParams { [JsonProperty] public Diagnostic[] diagnostics; } + + // + // Summary: + // A class representing a change that can be performed in code. A CodeAction must + // either set "edit" or "command". + // If both are supplied, the edit will be applied first, then the command will be executed. + [Serializable] + public sealed class CodeAction { + // + // Summary: + // Gets or sets the human readable title for this code action. + public string title; + // + // The kind of the code action. + // + // Used to filter code actions. + // + public string kind; + // + // The diagnostics that this code action resolves. + // + public Diagnostic[] diagnostics; + // + // Summary: + // Gets or sets the workspace edit that this code action performs. + public WorkspaceEdit edit; + // + // Summary: + // Gets or sets the command that this code action executes. + public Command command; + } } diff --git a/src/LanguageServer/Impl/Protocol/Diagnostic.cs b/src/LanguageServer/Impl/Protocol/Diagnostic.cs index 68dcac78b..9b1913792 100644 --- a/src/LanguageServer/Impl/Protocol/Diagnostic.cs +++ b/src/LanguageServer/Impl/Protocol/Diagnostic.cs @@ -51,4 +51,76 @@ public enum DiagnosticSeverity : int { Information = 3, Hint = 4 } + + // + // The kind of a code action. + // + // Kinds are a hierarchical list of identifiers separated by `.`, e.g. `"refactor.extract.function"`. + // + // The set of kinds is open and client needs to announce the kinds it supports to the server during + // initialization. + // + // + // A set of predefined code action kinds + // + public static class CodeActionKind { + // + // Empty kind. + // + public const string Empty = ""; + // + // Base kind for quickfix actions: 'quickfix' + // + public const string QuickFix = "quickfix"; + // + // Base kind for refactoring actions: 'refactor' + // + public const string Refactor = "refactor"; + // + // Base kind for refactoring extraction actions: 'refactor.extract' + // + // Example extract actions: + // + // - Extract method + // - Extract function + // - Extract variable + // - Extract interface from class + // - ... + // + public const string RefactorExtract = "refactor.extract"; + // + // Base kind for refactoring inline actions: 'refactor.inline' + // + // Example inline actions: + // + // - Inline function + // - Inline variable + // - Inline constant + // - ... + // + public const string RefactorInline = "refactor.inline"; + // + // Base kind for refactoring rewrite actions: 'refactor.rewrite' + // + // Example rewrite actions: + // + // - Convert JavaScript function to class + // - Add or remove parameter + // - Encapsulate field + // - Make method static + // - Move method to base class + // - ... + // + public const string RefactorRewrite = "refactor.rewrite"; + // + // Base kind for source actions: `source` + // + // Source code actions apply to the entire file. + // + public const string Source = "source"; + // + // Base kind for an organize imports source action: `source.organizeImports` + // + public const string SourceOrganizeImports = "source.organizeImports"; + } } diff --git a/src/LanguageServer/Impl/Protocol/Messages.cs b/src/LanguageServer/Impl/Protocol/Messages.cs index ab253a760..35c0f6f2f 100644 --- a/src/LanguageServer/Impl/Protocol/Messages.cs +++ b/src/LanguageServer/Impl/Protocol/Messages.cs @@ -189,13 +189,17 @@ public struct CodeActionParams { [Serializable] public sealed class CodeActionContext { + // + // An array of diagnostics. + // public Diagnostic[] diagnostics; - - /// - /// The intended version that diagnostic locations apply to. The request may - /// fail if the server cannot map correctly. - /// - public int? _version; + // + // Requested kind of actions to return. + // + // Actions not of this kind are filtered out by the client before being shown. So servers + // can omit computing them. + // + public string[] only; } [Serializable] diff --git a/src/LanguageServer/Impl/Python-Language-Server.nuspec b/src/LanguageServer/Impl/Python-Language-Server.nuspec index 6fb9ba44f..6986ab2b2 100644 --- a/src/LanguageServer/Impl/Python-Language-Server.nuspec +++ b/src/LanguageServer/Impl/Python-Language-Server.nuspec @@ -6,12 +6,13 @@ Python Language Server Microsoft Microsoft + https://github.com/Microsoft/python-language-server/blob/master/LICENSE https://github.com/Microsoft/python-language-server false Microsoft Python Language Server Summary of changes made in this release of the package. - Copyright (c) 2018-2019 + © Microsoft Corporation. All rights reserved. Python;VS Code diff --git a/src/LanguageServer/Impl/Resources.Designer.cs b/src/LanguageServer/Impl/Resources.Designer.cs index b413d0dde..1315808f6 100644 --- a/src/LanguageServer/Impl/Resources.Designer.cs +++ b/src/LanguageServer/Impl/Resources.Designer.cs @@ -105,6 +105,24 @@ internal static string Error_InvalidCachePath { } } + /// + /// Looks up a localized string similar to import locally '{0}'. + /// + internal static string ImportLocally { + get { + return ResourceManager.GetString("ImportLocally", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to Indexing libraries.. + /// + internal static string IndexingLibraries { + get { + return ResourceManager.GetString("IndexingLibraries", resourceCulture); + } + } + /// /// Looks up a localized string similar to Initializing for generic interpreter. /// @@ -168,6 +186,15 @@ internal static string RenameVariable_CannotRename { } } + /// + /// Looks up a localized string similar to No interpreter specified, using Python from PATH.. + /// + internal static string UsingPythonFromPATH { + get { + return ResourceManager.GetString("UsingPythonFromPATH", resourceCulture); + } + } + /// /// Looks up a localized string similar to Workspace root: {0}. /// diff --git a/src/LanguageServer/Impl/Resources.resx b/src/LanguageServer/Impl/Resources.resx index e060c233d..92802772f 100644 --- a/src/LanguageServer/Impl/Resources.resx +++ b/src/LanguageServer/Impl/Resources.resx @@ -132,6 +132,12 @@ Specified cache folder does not exist. Switching to default. + + import locally '{0}' + + + Indexing libraries. + Initializing for generic interpreter @@ -153,6 +159,9 @@ Cannot rename + + No interpreter specified, using Python from PATH. + Workspace root: {0} diff --git a/src/LanguageServer/Impl/Sources/DefinitionSource.cs b/src/LanguageServer/Impl/Sources/DefinitionSource.cs index 56f1e8a1a..e104afba4 100644 --- a/src/LanguageServer/Impl/Sources/DefinitionSource.cs +++ b/src/LanguageServer/Impl/Sources/DefinitionSource.cs @@ -14,6 +14,7 @@ // permissions and limitations under the License. using System; +using System.IO; using System.Linq; using Microsoft.Python.Analysis; using Microsoft.Python.Analysis.Analyzer; @@ -123,6 +124,8 @@ private Reference HandleFromImport(IDocumentAnalysis analysis, SourceLocation lo case ImplicitPackageImport packageImport: module = mres.GetImportedModule(packageImport.FullName); break; + case ImportNotFound _: + return null; } // Are we in the module name (i.e. A in 'from A import B')? @@ -140,7 +143,7 @@ private Reference HandleFromImport(IDocumentAnalysis analysis, SourceLocation lo // Are we in the member name part (ie. B in 'from A import B')? // Handle 'from A import B' similar to 'import A.B' - var partReference = FindModulePartReference(statement.Names, expr, module, out definingMember); + var partReference = FindModulePartReference(statement.Names, statement.AsNames, expr, module, out definingMember); if (partReference != null) { return partReference; } @@ -163,9 +166,24 @@ private Reference HandleFromImport(IDocumentAnalysis analysis, SourceLocation lo return null; } - private static Reference FindModulePartReference(ImmutableArray names, Node expr, IPythonModule module, out ILocatedMember definingMember) { + private static Reference FindModulePartReference(ImmutableArray names, ImmutableArray asNames, Node expr, IPythonModule module, out ILocatedMember definingMember) { definingMember = null; - var part = names.FirstOrDefault(x => x.IndexSpan.Start <= expr.StartIndex && x.IndexSpan.Start <= expr.EndIndex); + NameExpression part = null; + for (var i = 0; i < names.Count; i++) { + var name = names[i]; + var asName = i < asNames.Count ? asNames[i] : null; + + if (expr.StartIndex <= name.IndexSpan.Start && expr.EndIndex >= name.IndexSpan.End) { + part = name; + break; + } + + if (asName != null && expr.StartIndex <= asName.IndexSpan.Start && expr.EndIndex >= asName.IndexSpan.End) { + part = name; + break; + } + } + if (part != null) { var definition = module.Analysis.GlobalScope.Variables[part.Name]?.Definition; if (definition != null) { @@ -187,12 +205,12 @@ private Reference HandleImport(IDocumentAnalysis analysis, ImportStatement state } var reference = FindModuleNamePartReference(analysis, statement.Names, expr, out definingMember); - if(reference != null) { + if (reference != null) { return reference; } // Import A as B - var asName = statement.AsNames.FirstOrDefault(n => n.IndexSpan.Start <= expr.StartIndex && n.IndexSpan.Start <= expr.EndIndex); + var asName = statement.AsNames.ExcludeDefault().FirstOrDefault(n => n.IndexSpan.Start <= expr.StartIndex && n.IndexSpan.Start <= expr.EndIndex); if (asName != null) { var value = analysis.ExpressionEvaluator.GetValueFromExpression(asName); if (!value.IsUnknown()) { @@ -247,7 +265,12 @@ private Reference TryFromVariable(string name, IDocumentAnalysis analysis, Sourc definingMember = null; var m = analysis.ExpressionEvaluator.LookupNameInScopes(name, out var scope, LookupOptions.All); - if (m == null || scope.Module.ModuleType == ModuleType.Builtins || !(scope.Variables[name] is IVariable v)) { + if(m == null || scope == null || scope.Module.ModuleType == ModuleType.Builtins) { + return null; + } + + var v = scope.Variables[name]; + if (v == null || (v.Source == VariableSource.Import && v.IsUnknown())) { return null; } @@ -286,7 +309,19 @@ private Reference FromMemberExpression(MemberExpression mex, IDocumentAnalysis a return FromMember(v1); } break; - + case IPythonSuperType cls: // TODO: move mro IPythontypes or make IPythonClassType a child of IPythonSuperType + // Data members may be PythonInstances which do not track their declaration location. + // In this case we'll try looking up the respective variable instead according to Mro. + foreach (var b in cls.Mro.OfType()) { + using (eval.OpenScope(b)) { + eval.LookupNameInScopes(mex.Name, out _, out var v2, LookupOptions.Local); + if (v2 != null) { + definingMember = v2; + return FromMember(v2); + } + } + } + break; case IPythonClassType cls: // Data members may be PythonInstances which do not track their declaration location. // In this case we'll try looking up the respective variable instead according to Mro. @@ -326,10 +361,14 @@ private Reference FromMember(IMember m) { return null; } - private bool CanNavigateToModule(Uri uri) { + public bool CanNavigateToModule(Uri uri) { if (uri == null) { return false; } + + if (!CanNavigateToPath(uri.LocalPath)) { + return false; + } var rdt = _services.GetService(); var doc = rdt.GetDocument(uri); // Allow navigation to modules not in RDT - most probably @@ -337,11 +376,17 @@ private bool CanNavigateToModule(Uri uri) { return doc == null || CanNavigateToModule(doc); } - private static bool CanNavigateToModule(IPythonModule m) - => m?.ModuleType == ModuleType.User || - m?.ModuleType == ModuleType.Stub || - m?.ModuleType == ModuleType.Package || - m?.ModuleType == ModuleType.Library || - m?.ModuleType == ModuleType.Specialized; + private static bool CanNavigateToModule(IPythonModule m) { + if(m == null || !CanNavigateToPath(m.FilePath)) { + return false; + } + return m.ModuleType == ModuleType.User || + m.ModuleType == ModuleType.Stub || + m.ModuleType == ModuleType.Package || + m.ModuleType == ModuleType.Library || + m.ModuleType == ModuleType.Specialized; + } + + private static bool CanNavigateToPath(string path) => Path.GetExtension(path) != ".exe"; } } diff --git a/src/LanguageServer/Impl/Sources/DocumentHighlightSource.cs b/src/LanguageServer/Impl/Sources/DocumentHighlightSource.cs new file mode 100644 index 000000000..afe1efd42 --- /dev/null +++ b/src/LanguageServer/Impl/Sources/DocumentHighlightSource.cs @@ -0,0 +1,64 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis; +using Microsoft.Python.Analysis.Documents; +using Microsoft.Python.Analysis.Modules; +using Microsoft.Python.Analysis.Types; +using Microsoft.Python.Core; +using Microsoft.Python.Core.IO; +using Microsoft.Python.Core.Text; +using Microsoft.Python.LanguageServer.Documents; +using Microsoft.Python.LanguageServer.Protocol; + +namespace Microsoft.Python.LanguageServer.Sources { + internal sealed class DocumentHighlightSource { + private const int DocumentHighlightAnalysisTimeout = 10000; + private readonly IServiceContainer _services; + + public DocumentHighlightSource(IServiceContainer services) { + _services = services; + } + + public async Task DocumentHighlightAsync(Uri uri, SourceLocation location, CancellationToken cancellationToken = default) { + if (uri == null) { + return Array.Empty(); + } + + var analysis = await Document.GetAnalysisAsync(uri, _services, DocumentHighlightAnalysisTimeout, cancellationToken); + var definitionSource = new DefinitionSource(_services); + + var definition = definitionSource.FindDefinition(analysis, location, out var definingMember); + if (definition == null || definingMember == null) { + return Array.Empty(); + } + + var rootDefinition = definingMember.GetRootDefinition(); + + var result = rootDefinition.References + .Where(r => r.DocumentUri.Equals(uri)) + .Select((r, i) => new DocumentHighlight { kind = (i == 0) ? DocumentHighlightKind.Write : DocumentHighlightKind.Read, range = r.Span }) + .ToArray(); + + return result; + } + } +} diff --git a/src/LanguageServer/Impl/Sources/DocumentationSource.cs b/src/LanguageServer/Impl/Sources/DocumentationSource.cs index 91ee205b0..de2b97583 100644 --- a/src/LanguageServer/Impl/Sources/DocumentationSource.cs +++ b/src/LanguageServer/Impl/Sources/DocumentationSource.cs @@ -13,57 +13,85 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. -using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; using Microsoft.Python.Analysis; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Core.Text; +using Microsoft.Python.Parsing.Ast; namespace Microsoft.Python.LanguageServer.Sources { internal abstract class DocumentationSource { - public string GetSignatureString(IPythonFunctionType ft, IPythonType self, out IndexSpan[] parameterSpans, int overloadIndex = 0, string name = null) { + public string GetSignatureString(IPythonFunctionType ft, IPythonType self, out (IndexSpan, IParameterInfo)[] parameterSpans, int overloadIndex = 0, string funcName = null, bool noReturn = false) { + funcName = funcName ?? ft.Name; var o = ft.Overloads[overloadIndex]; - var parameterStrings = GetFunctionParameters(ft, out var parameterNameLengths); - var returnDoc = o.GetReturnDocumentation(self); - var annString = string.IsNullOrEmpty(returnDoc) ? string.Empty : $" -> {returnDoc}"; + var spans = new List<(IndexSpan, IParameterInfo)>(); + var builder = new StringBuilder(funcName); + builder.Append('('); - // Calculate parameter spans - parameterSpans = new IndexSpan[parameterStrings.Length]; - name = name ?? ft.Name; - var offset = name.Length + 1; - for (var i = 0; i < parameterStrings.Length; i++) { - parameterSpans[i] = IndexSpan.FromBounds(offset, offset + parameterNameLengths[i]); - offset += parameterStrings[i].Length + 2; // name, - } + var addComma = false; + var addMarker = false; - var combinedParameterString = string.Join(", ", parameterStrings); - return $"{name}({combinedParameterString}){annString}"; - } + foreach (var p in o.Parameters.Skip(ft.IsStatic || ft.IsUnbound ? 0 : 1)) { + if (addComma) { + builder.Append(", "); + } else { + addComma = true; + } + + // Positional only markers are not included in the parameters, so keep track + // of whether or not positional only parameters have been seen then add the marker + // once the positional only section ends. + if (p.Kind == ParameterKind.PositionalOnly) { + addMarker = true; + } else if (addMarker && p.Kind != ParameterKind.PositionalOnly) { + builder.Append("/, "); + addMarker = false; + } + + switch (p.Kind) { + case ParameterKind.List: + builder.Append("*"); + if (string.IsNullOrEmpty(p.Name)) { + continue; + } + break; + case ParameterKind.Dictionary: + builder.Append("**"); + break; + } - /// - /// Constructs parameter strings that include parameter name, type and default value - /// like 'x: int' or 'a = None'. Returns collection of parameter strings and - /// the respective lengths of names for rendering in bold (as current parameter). - /// - private string[] GetFunctionParameters(IPythonFunctionType ft, out int[] parameterNameLengths, int overloadIndex = 0) { - var o = ft.Overloads[overloadIndex]; // TODO: display all? - var skip = ft.IsStatic || ft.IsUnbound ? 0 : 1; + var name = p.Name ?? string.Empty; + spans.Add((new IndexSpan(builder.Length, name.Length), p)); + builder.Append(name); - var count = Math.Max(0, o.Parameters.Count - skip); - var parameters = new string[count]; - parameterNameLengths = new int[count]; - for (var i = skip; i < o.Parameters.Count; i++) { - string paramString; - var p = o.Parameters[i]; if (!string.IsNullOrEmpty(p.DefaultValueString)) { - paramString = $"{p.Name}={p.DefaultValueString}"; - } else { - paramString = p.Type.IsUnknown() ? p.Name : $"{p.Name}: {p.Type.Name}"; + builder.Append('='); + builder.Append(p.DefaultValueString); + } else if (!p.Type.IsUnknown()) { + builder.Append(": "); + builder.Append(p.Type.Name); + } + } + + if (addMarker) { + builder.Append(", /"); + } + + builder.Append(')'); + + if (!noReturn) { + var returnDoc = o.GetReturnDocumentation(self); + if (!string.IsNullOrWhiteSpace(returnDoc)) { + builder.Append(" -> "); + builder.Append(returnDoc); } - parameters[i - skip] = paramString; - parameterNameLengths[i - skip] = p.Name.Length; } - return parameters; + + parameterSpans = spans.ToArray(); + return builder.ToString(); } } } diff --git a/src/LanguageServer/Impl/Sources/HoverSource.cs b/src/LanguageServer/Impl/Sources/HoverSource.cs index 22ed0aeb5..4f843714e 100644 --- a/src/LanguageServer/Impl/Sources/HoverSource.cs +++ b/src/LanguageServer/Impl/Sources/HoverSource.cs @@ -54,27 +54,27 @@ public Hover GetHover(IDocumentAnalysis analysis, SourceLocation location) { var eval = analysis.ExpressionEvaluator; switch (statement) { case FromImportStatement fi when node is NameExpression nex: { - var contents = HandleFromImport(fi, location, hoverScopeStatement, analysis); - if (contents != null) { - return new Hover { - contents = contents, - range = range - }; - } - - break; + var contents = HandleFromImport(fi, location, hoverScopeStatement, analysis); + if (contents != null) { + return new Hover { + contents = contents, + range = range + }; } - case ImportStatement imp: { - var contents = HandleImport(imp, location, hoverScopeStatement, analysis); - if (contents != null) { - return new Hover { - contents = contents, - range = range - }; - } - break; + break; + } + case ImportStatement imp: { + var contents = HandleImport(imp, location, hoverScopeStatement, analysis); + if (contents != null) { + return new Hover { + contents = contents, + range = range + }; } + + break; + } } IMember value; @@ -96,13 +96,13 @@ public Hover GetHover(IDocumentAnalysis analysis, SourceLocation location) { IVariable variable = null; if (expr is NameExpression nex) { - analysis.ExpressionEvaluator.LookupNameInScopes(nex.Name, out _, out variable, LookupOptions.All); + eval.LookupNameInScopes(nex.Name, out _, out variable, LookupOptions.All); if (IsInvalidClassMember(variable, hoverScopeStatement, location.ToIndex(analysis.Ast))) { return null; } } - value = variable?.Value ?? analysis.ExpressionEvaluator.GetValueFromExpression(expr, LookupOptions.All); + value = variable?.Value ?? eval.GetValueFromExpression(expr, LookupOptions.All); type = value?.GetPythonType(); if (type == null) { return null; @@ -122,7 +122,7 @@ public Hover GetHover(IDocumentAnalysis analysis, SourceLocation location) { // In case of a member expression get the target since if we end up with method // of a generic class, the function will need specific type to determine its return // value correctly. I.e. in x.func() we need to determine type of x (self for func). - var v = analysis.ExpressionEvaluator.GetValueFromExpression(mex.Target); + var v = eval.GetValueFromExpression(mex.Target); self = v?.GetPythonType(); } @@ -165,7 +165,7 @@ private bool IsInvalidClassMember(IVariable v, ScopeStatement scope, int hoverPo } switch (v.Value) { case IPythonClassType cls when cls.ClassDefinition == scope: - return hoverPosition > cls.ClassDefinition.HeaderIndex; + return hoverPosition > cls.ClassDefinition.HeaderIndex; case IPythonFunctionType ft when ft.FunctionDefinition == scope: return hoverPosition > ft.FunctionDefinition.HeaderIndex; case IPythonPropertyType prop when prop.FunctionDefinition == scope: diff --git a/src/LanguageServer/Impl/Sources/MarkdownDocumentationSource.cs b/src/LanguageServer/Impl/Sources/MarkdownDocumentationSource.cs index 632e4126c..63a9a7e56 100644 --- a/src/LanguageServer/Impl/Sources/MarkdownDocumentationSource.cs +++ b/src/LanguageServer/Impl/Sources/MarkdownDocumentationSource.cs @@ -13,6 +13,7 @@ // See the Apache Version 2.0 License for specific language governing // permissions and limitations under the License. +using System.Linq; using Microsoft.Python.Analysis; using Microsoft.Python.Analysis.Types; using Microsoft.Python.Analysis.Values; @@ -23,7 +24,7 @@ namespace Microsoft.Python.LanguageServer.Sources { internal sealed class MarkdownDocumentationSource : DocumentationSource, IDocumentationSource { public InsertTextFormat DocumentationFormat => InsertTextFormat.PlainText; - public MarkupContent GetHover(string name, IMember member, IPythonType self) { + public MarkupContent GetHover(string name, IMember member, IPythonType self, bool includeClassInit = false) { // We need to tell between instance and type. var type = member.GetPythonType(); if (type.IsUnknown()) { @@ -41,16 +42,30 @@ public MarkupContent GetHover(string name, IMember member, IPythonType self) { var typeDoc = !string.IsNullOrEmpty(type.Documentation) ? $"\n---\n{type.MarkdownDoc()}" : string.Empty; switch (type) { case IPythonPropertyType prop: - text = GetPropertyHoverString(prop); + text = GetPropertyString(prop); break; case IPythonFunctionType ft: - text = GetFunctionHoverString(ft, self); + text = GetFunctionString(ft, self); break; case IPythonClassType cls: var clsDoc = !string.IsNullOrEmpty(cls.Documentation) ? $"\n---\n{cls.MarkdownDoc()}" : string.Empty; - text = $"```\nclass {cls.Name}\n```{clsDoc}"; + + string className; + var sig = string.Empty; + + if (includeClassInit) { + className = cls.Name; + var init = cls.TryGetFunctionType(); + if (init != null) { + sig = GetSignatureString(init, null, out var _, 0, "", true); + } + } else { + className = "class " + cls.Name; + } + + text = $"```\n{className}{sig}\n```{clsDoc}"; break; case IPythonModule mod: @@ -67,7 +82,7 @@ public MarkupContent GetHover(string name, IMember member, IPythonType self) { }; } - public MarkupContent FormatDocumentation(string documentation) + public MarkupContent FormatDocumentation(string documentation) => new MarkupContent { kind = MarkupKind.Markdown, value = DocstringConverter.ToMarkdown(documentation) }; public MarkupContent FormatParameterDocumentation(IParameterInfo parameter) { @@ -79,13 +94,13 @@ public MarkupContent FormatParameterDocumentation(IParameterInfo parameter) { return new MarkupContent { kind = MarkupKind.Markdown, value = text }; } - private string GetPropertyHoverString(IPythonPropertyType prop, int overloadIndex = 0) { + private string GetPropertyString(IPythonPropertyType prop) { var decTypeString = prop.DeclaringType != null ? $"{prop.DeclaringType.Name}." : string.Empty; var propDoc = !string.IsNullOrEmpty(prop.Documentation) ? $"\n---\n{prop.MarkdownDoc()}" : string.Empty; return $"```\n{decTypeString}\n```{propDoc}"; } - private string GetFunctionHoverString(IPythonFunctionType ft, IPythonType self, int overloadIndex = 0) { + private string GetFunctionString(IPythonFunctionType ft, IPythonType self, int overloadIndex = 0) { var sigString = GetSignatureString(ft, self, out _, overloadIndex); var decTypeString = ft.DeclaringType != null ? $"{ft.DeclaringType.Name}." : string.Empty; var funcDoc = !string.IsNullOrEmpty(ft.Documentation) ? $"\n---\n{ft.MarkdownDoc()}" : string.Empty; diff --git a/src/LanguageServer/Impl/Sources/PlainTextDocumentationSource.cs b/src/LanguageServer/Impl/Sources/PlainTextDocumentationSource.cs index 3a3e9c9ef..562df662c 100644 --- a/src/LanguageServer/Impl/Sources/PlainTextDocumentationSource.cs +++ b/src/LanguageServer/Impl/Sources/PlainTextDocumentationSource.cs @@ -23,7 +23,7 @@ namespace Microsoft.Python.LanguageServer.Sources { internal sealed class PlainTextDocumentationSource : DocumentationSource, IDocumentationSource { public InsertTextFormat DocumentationFormat => InsertTextFormat.PlainText; - public MarkupContent GetHover(string name, IMember member, IPythonType self) { + public MarkupContent GetHover(string name, IMember member, IPythonType self, bool includeClassInit = false) { // We need to tell between instance and type. var type = member.GetPythonType(); if (type.IsUnknown()) { @@ -50,7 +50,21 @@ public MarkupContent GetHover(string name, IMember member, IPythonType self) { case IPythonClassType cls: var clsDoc = !string.IsNullOrEmpty(cls.Documentation) ? $"\n\n{cls.PlaintextDoc()}" : string.Empty; - text = $"class {cls.Name}{clsDoc}"; + + string className; + var sig = string.Empty; + + if (includeClassInit) { + className = cls.Name; + var init = cls.GetMember("__init__"); + if (init != null) { + sig = GetSignatureString(init, null, out var _, 0, "", true); + } + } else { + className = "class " + cls.Name; + } + + text = $"{className}{sig}{clsDoc}"; break; case IPythonModule mod: @@ -67,7 +81,7 @@ public MarkupContent GetHover(string name, IMember member, IPythonType self) { }; } - public MarkupContent FormatDocumentation(string documentation) + public MarkupContent FormatDocumentation(string documentation) => new MarkupContent { kind = MarkupKind.PlainText, value = documentation }; public MarkupContent FormatParameterDocumentation(IParameterInfo parameter) { diff --git a/src/LanguageServer/Impl/Sources/QuickFixCodeActionSource.cs b/src/LanguageServer/Impl/Sources/QuickFixCodeActionSource.cs new file mode 100644 index 000000000..97e4b712b --- /dev/null +++ b/src/LanguageServer/Impl/Sources/QuickFixCodeActionSource.cs @@ -0,0 +1,77 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis; +using Microsoft.Python.Analysis.Diagnostics; +using Microsoft.Python.Core; +using Microsoft.Python.Core.Collections; +using Microsoft.Python.LanguageServer.CodeActions; +using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.LanguageServer.Sources { + internal sealed partial class QuickFixCodeActionSource { + private static readonly ImmutableArray _codeActionProviders = + ImmutableArray.Create(MissingImportCodeActionProvider.Instance); + + private readonly IServiceContainer _services; + + public QuickFixCodeActionSource(IServiceContainer services) { + _services = services; + } + + public async Task GetCodeActionsAsync(IDocumentAnalysis analysis, CodeActionSettings settings, Diagnostic[] diagnostics, CancellationToken cancellationToken) { + var results = new List(); + + foreach (var diagnostic in GetMatchingDiagnostics(analysis, diagnostics, cancellationToken)) { + foreach (var codeActionProvider in _codeActionProviders) { + cancellationToken.ThrowIfCancellationRequested(); + + if (codeActionProvider.FixableDiagnostics.Any(code => code == diagnostic.ErrorCode)) { + results.AddRange(await codeActionProvider.GetCodeActionsAsync(analysis, settings, diagnostic, cancellationToken)); + } + } + } + + return results.ToArray(); + } + + private IEnumerable GetMatchingDiagnostics(IDocumentAnalysis analysis, Diagnostic[] diagnostics, CancellationToken cancellationToken) { + var diagnosticService = _services.GetService(); + + // we assume diagnostic service has the latest results + if (diagnosticService == null || !diagnosticService.Diagnostics.TryGetValue(analysis.Document.Uri, out var latestDiagnostics)) { + yield break; + } + + foreach (var diagnostic in latestDiagnostics) { + cancellationToken.ThrowIfCancellationRequested(); + + if (diagnostics.Any(d => AreEqual(d, diagnostic))) { + yield return diagnostic; + } + } + + bool AreEqual(Diagnostic diagnostic1, DiagnosticsEntry diagnostic2) { + return diagnostic1.code == diagnostic2.ErrorCode && + diagnostic1.range.ToSourceSpan() == diagnostic2.SourceSpan; + } + } + } +} diff --git a/src/LanguageServer/Impl/Sources/RefactoringCodeActionSource.cs b/src/LanguageServer/Impl/Sources/RefactoringCodeActionSource.cs new file mode 100644 index 000000000..1053b6369 --- /dev/null +++ b/src/LanguageServer/Impl/Sources/RefactoringCodeActionSource.cs @@ -0,0 +1,48 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis; +using Microsoft.Python.Core; +using Microsoft.Python.Core.Collections; +using Microsoft.Python.Core.Text; +using Microsoft.Python.LanguageServer.CodeActions; +using Microsoft.Python.LanguageServer.Protocol; + +namespace Microsoft.Python.LanguageServer.Sources { + internal sealed partial class RefactoringCodeActionSource { + private static readonly ImmutableArray _codeActionProviders = + ImmutableArray.Empty; + + private readonly IServiceContainer _services; + + public RefactoringCodeActionSource(IServiceContainer services) { + _services = services; + } + + public async Task GetCodeActionsAsync(IDocumentAnalysis analysis, CodeActionSettings settings, Range range, CancellationToken cancellationToken) { + var results = new List(); + foreach (var codeActionProvider in _codeActionProviders) { + cancellationToken.ThrowIfCancellationRequested(); + + results.AddRange(await codeActionProvider.GetCodeActionsAsync(analysis, settings, range, cancellationToken)); + } + + return results.ToArray(); + } + } +} diff --git a/src/LanguageServer/Impl/Sources/ReferenceSource.cs b/src/LanguageServer/Impl/Sources/ReferenceSource.cs index ca846f513..a4cdcc5b3 100644 --- a/src/LanguageServer/Impl/Sources/ReferenceSource.cs +++ b/src/LanguageServer/Impl/Sources/ReferenceSource.cs @@ -44,24 +44,35 @@ public ReferenceSource(IServiceContainer services) { } public async Task FindAllReferencesAsync(Uri uri, SourceLocation location, ReferenceSearchOptions options, CancellationToken cancellationToken = default) { - if (uri != null) { - var analysis = await Document.GetAnalysisAsync(uri, _services, FindReferencesAnalysisTimeout, cancellationToken); + if (uri == null) { + return Array.Empty(); + } + var analysis = await Document.GetAnalysisAsync(uri, _services, FindReferencesAnalysisTimeout, cancellationToken); - var definitionSource = new DefinitionSource(_services); - var definition = definitionSource.FindDefinition(analysis, location, out var definingMember); - if (definition == null) { - return Array.Empty(); - } + var definitionSource = new DefinitionSource(_services); + var definition = definitionSource.FindDefinition(analysis, location, out var definingMember); + if (definition == null) { + return Array.Empty(); + } - var rootDefinition = definingMember.GetRootDefinition(); - var name = definingMember.GetName(); + var rootDefinition = definingMember.GetRootDefinition(); + if (rootDefinition == null) { + return Array.Empty(); + } + + var name = definingMember.GetName(); + var moduleUri = rootDefinition.Definition?.DocumentUri; + if (moduleUri == null) { + return Array.Empty(); + } + + var rdt = _services.GetService(); + var declaringModule = rdt.GetDocument(moduleUri); - // If it is an implicitly declared variable, such as function or a class - // then the location is invalid and the module is null. Use current module. - var declaringModule = rootDefinition?.DeclaringModule ?? analysis.Document; - if (!string.IsNullOrEmpty(name) && (declaringModule.ModuleType == ModuleType.User || options == ReferenceSearchOptions.All)) { - return await FindAllReferencesAsync(name, declaringModule, rootDefinition, location, definitionSource, cancellationToken); - } + // If it is an implicitly declared variable, such as function or a class + // then the location is invalid and the module is null. Use current module. + if (!string.IsNullOrEmpty(name) && (declaringModule.ModuleType == ModuleType.User || options == ReferenceSearchOptions.All)) { + return await FindAllReferencesAsync(name, declaringModule, rootDefinition, location, definitionSource, cancellationToken); } return Array.Empty(); } diff --git a/src/LanguageServer/Impl/Sources/SignatureSource.cs b/src/LanguageServer/Impl/Sources/SignatureSource.cs index dbde79d28..5675a7d83 100644 --- a/src/LanguageServer/Impl/Sources/SignatureSource.cs +++ b/src/LanguageServer/Impl/Sources/SignatureSource.cs @@ -64,31 +64,20 @@ public SignatureHelp GetSignature(IDocumentAnalysis analysis, SourceLocation loc } } - IPythonFunctionType ft; - - if (value is IPythonClassType cls) { - ft = cls.GetMember("__init__"); - } else { - ft = value?.GetPythonType(); - } - + var ft = value.TryGetFunctionType(); if (ft == null) { return null; } - var skip = ft.IsStatic || ft.IsUnbound ? 0 : 1; - var signatures = new SignatureInformation[ft.Overloads.Count]; for (var i = 0; i < ft.Overloads.Count; i++) { var o = ft.Overloads[i]; var signatureLabel = _docSource.GetSignatureString(ft, selfType, out var parameterSpans, i, name); - var visibleParameterCount = o.Parameters.Count - skip; - var parameterInfo = new ParameterInformation[visibleParameterCount]; - for (var j = 0; j < visibleParameterCount; j++) { - var p = o.Parameters[j + skip]; - var ps = parameterSpans[j]; + var parameterInfo = new ParameterInformation[parameterSpans.Length]; + for (var j = 0; j < parameterSpans.Length; j++) { + var (ps, p) = parameterSpans[j]; parameterInfo[j] = new ParameterInformation { label = _labelOffsetSupport ? new[] { ps.Start, ps.End } : (object)p.Name, diff --git a/src/LanguageServer/Impl/Telemetry/RequestTimer.cs b/src/LanguageServer/Impl/Telemetry/RequestTimer.cs index 3cb173e8c..5dab54064 100644 --- a/src/LanguageServer/Impl/Telemetry/RequestTimer.cs +++ b/src/LanguageServer/Impl/Telemetry/RequestTimer.cs @@ -24,7 +24,6 @@ internal class RequestTimer { private const int MaxEvents = 10; private readonly ITelemetryService _telemetryService; - private readonly Dictionary _events = new Dictionary(); private readonly object _lock = new object(); @@ -87,24 +86,25 @@ public void Dispose() { } _stopwatch.Stop(); - - var e = Telemetry.CreateEvent("rpc.request"); - e.Properties["method"] = _method; - e.Measurements["elapsedMs"] = _stopwatch.Elapsed.TotalMilliseconds; - - if (_extraProperties != null) { - foreach (var (key, value) in _extraProperties) { - e.Properties[key] = value; + if (_telemetryService != null) { + var e = Telemetry.CreateEvent("rpc.request"); + e.Properties["method"] = _method; + e.Measurements["elapsedMs"] = _stopwatch.Elapsed.TotalMilliseconds; + + if (_extraProperties != null) { + foreach (var (key, value) in _extraProperties) { + e.Properties[key] = value; + } } - } - if (_extraMeasures != null) { - foreach (var (key, value) in _extraMeasures) { - e.Measurements[key] = value; + if (_extraMeasures != null) { + foreach (var (key, value) in _extraMeasures) { + e.Measurements[key] = value; + } } - } - _telemetryService.SendTelemetryAsync(e).DoNotWait(); + _telemetryService.SendTelemetryAsync(e).DoNotWait(); + } } } } diff --git a/src/LanguageServer/Impl/Utilities/UniqueNameGenerator.cs b/src/LanguageServer/Impl/Utilities/UniqueNameGenerator.cs new file mode 100644 index 000000000..752c9f569 --- /dev/null +++ b/src/LanguageServer/Impl/Utilities/UniqueNameGenerator.cs @@ -0,0 +1,114 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Analysis; +using Microsoft.Python.Analysis.Analyzer.Expressions; +using Microsoft.Python.Parsing.Ast; +using Microsoft.Python.Parsing.Extensions; + +namespace Microsoft.Python.LanguageServer.Utilities { + /// + /// Generate unique identifier based on given context + /// + internal class UniqueNameGenerator { + private readonly IDocumentAnalysis _analysis; + private readonly ScopeStatement _scope; + private readonly bool _uniqueInModule; + + public static string Generate(IDocumentAnalysis analysis, int position, string name) { + var generator = new UniqueNameGenerator(analysis, position); + return generator.Generate(name); + } + + public static string Generate(IDocumentAnalysis analysis, string name) { + var generator = new UniqueNameGenerator(analysis, position: -1); + return generator.Generate(name); + } + + public UniqueNameGenerator(IDocumentAnalysis analysis, int position) { + _analysis = analysis; + _uniqueInModule = position < 0; + + if (!_uniqueInModule) { + var finder = new ExpressionFinder(analysis.Ast, new FindExpressionOptions() { Names = true }); + finder.Get(position, position, out _, out _, out _scope); + } + } + + public string Generate(string name) { + // for now, there is only 1 new name rule which is just incrementing count at the end. + int count = 0; + Func getNextName = () => { + return $"{name}{++count}"; + }; + + // for now, everything is fixed. and there is no knob to control what to consider when + // creating unique name and how to create new name if there is a conflict + if (_uniqueInModule) { + return GenerateModuleWideUniqueName(name, getNextName); + } else { + return GenerateContextBasedUniqueName(name, getNextName); + } + } + + private string GenerateModuleWideUniqueName(string name, Func getNextName) { + // think of a better way to do this. + var leafScopes = GetLeafScopes(_analysis.Ast.ChildNodesDepthFirst().OfType()); + + while (true) { + if (!leafScopes.Any(s => NameExist(name, s))) { + return name; + } + + name = getNextName(); + } + } + + private HashSet GetLeafScopes(IEnumerable scopes) { + var set = scopes.ToHashSet(); + foreach (var scope in set.ToList()) { + if (scope.Parent != null) { + set.Remove(scope.Parent); + } + } + + return set; + } + + private bool NameExist(string name, ScopeStatement scope) { + var eval = _analysis.ExpressionEvaluator; + using (eval.OpenScope(_analysis.Document, scope)) { + return eval.LookupNameInScopes(name, Analysis.Analyzer.LookupOptions.All) != null; + } + } + + private string GenerateContextBasedUniqueName(string name, Func getNextName) { + var eval = _analysis.ExpressionEvaluator; + using (eval.OpenScope(_analysis.Document, _scope)) { + while (true) { + var member = eval.LookupNameInScopes(name, Analysis.Analyzer.LookupOptions.All); + if (member == null) { + return name; + } + + name = getNextName(); + } + } + } + } +} diff --git a/src/LanguageServer/Test/CacheManagementTests.cs b/src/LanguageServer/Test/CacheManagementTests.cs new file mode 100644 index 000000000..02ffae0a5 --- /dev/null +++ b/src/LanguageServer/Test/CacheManagementTests.cs @@ -0,0 +1,83 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Python.Analysis.Caching; +using Microsoft.Python.Core.IO; +using Microsoft.Python.Core.Services; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NSubstitute; +using StreamJsonRpc; +using TestUtilities; + +namespace Microsoft.Python.LanguageServer.Tests { + [TestClass] + public class CacheManagementTests : LanguageServerTestBase { + public TestContext TestContext { get; set; } + + [TestInitialize] + public void TestInitialize() + => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + + [TestCleanup] + public void Cleanup() => TestEnvironmentImpl.TestCleanup(); + + [TestMethod, Priority(0)] + public Task DeleteAnalysisCache() + => RunTest(sm => { + using (var s = new Implementation.Server(sm)) { + s.ClearAnalysisCache(); + return Task.CompletedTask; + } + }); + + [TestMethod, Priority(0)] + public Task LsDeleteAnalysisCache() + => RunTest(sm => { + using (var ls = new Implementation.LanguageServer()) + using (var ms = new MemoryStream()) { + ls.Start(sm, new JsonRpc(ms)); + return ls.ClearAnalysisCache(CancellationToken.None); + } + }); + + public async Task RunTest(Func test) { + using (var sm = new ServiceManager()) { + var fs = Substitute.For(); + sm.AddService(fs); + + const string baseName = "analysis.v"; + SetupDirectories(fs, new[] { $"{baseName}1", $"{baseName}3" }); + + var mdc = Substitute.For(); + mdc.CacheFolder.Returns(c => "CacheFolder"); + sm.AddService(mdc); + + await test(sm); + fs.Received().DeleteDirectory($"{baseName}1", true); + fs.Received().DeleteDirectory($"{baseName}3", true); + } + } + private static void SetupDirectories(IFileSystem fs, string[] names) { + fs.GetFileSystemEntries(Arg.Any(), Arg.Any(), Arg.Any()) + .ReturnsForAnyArgs(c => names); + fs.GetFileAttributes(Arg.Any()) + .ReturnsForAnyArgs(c => FileAttributes.Directory); + } + } +} diff --git a/src/LanguageServer/Test/CompletionTests.cs b/src/LanguageServer/Test/CompletionTests.cs index 625eba996..28a5fb805 100644 --- a/src/LanguageServer/Test/CompletionTests.cs +++ b/src/LanguageServer/Test/CompletionTests.cs @@ -16,6 +16,7 @@ using System; using System.IO; using System.Linq; +using System.Threading; using System.Threading.Tasks; using FluentAssertions; using Microsoft.Python.Analysis; @@ -58,7 +59,7 @@ def method(self): "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(8, 1)); comps.Should().HaveLabels("C", "x", "y", "while", "for"); } @@ -70,7 +71,7 @@ public async Task StringMembers() { x. "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(3, 3)); comps.Should().HaveLabels(@"isupper", @"capitalize", @"split"); } @@ -82,7 +83,7 @@ import datetime datetime.datetime. "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(3, 19)); comps.Should().HaveLabels("now", @"tzinfo", @"ctime"); } @@ -97,7 +98,7 @@ def method1(self): pass ABCDE.me "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(5, 4)); comps.Should().HaveLabels(@"ABCDE"); @@ -115,7 +116,7 @@ class oar(list): pass "; var analysis = await GetAnalysisAsync(code, version); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 9)); result.Should().HaveItem("append") @@ -126,12 +127,16 @@ class oar(list): [TestMethod] public async Task OverrideInit3X() { const string code = @" -class Test(): +class A: + def __init__(self, *args, **kwargs): + pass + +class Test(A): def __ "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); - var result = cs.GetCompletions(analysis, new SourceLocation(3, 10)); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var result = cs.GetCompletions(analysis, new SourceLocation(7, 10)); result.Should().HaveItem("__init__") .Which.Should().HaveInsertText($"__init__(self, *args, **kwargs):{Environment.NewLine} super().__init__(*args, **kwargs)") @@ -148,7 +153,7 @@ class oar(list): pass "; var analysis = await GetAnalysisAsync(code, version); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 9)); result.Should().HaveItem("append") @@ -167,7 +172,7 @@ class Test(A): def __ "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(7, 10)); result.Should().HaveItem("__init__") @@ -191,7 +196,7 @@ def fob(self): "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(4, 8)); result.Should().HaveItem("a"); } @@ -208,7 +213,7 @@ def oar(self, a): "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(4, 8)); result.Should().HaveItem("a"); } @@ -230,7 +235,7 @@ def oar(self): pass var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(11, 3)); result.Should().NotContainLabels("fob"); result.Should().HaveLabels("oar"); @@ -249,7 +254,7 @@ class B(A): def f"; var analysis = await GetAnalysisAsync(code, is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(7, 10)); result.Should() @@ -264,13 +269,13 @@ public async Task InRaise(bool is3X) { var version = is3X ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X; var analysis = await GetAnalysisAsync("raise ", version); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 7)); result.Should().HaveInsertTexts("Exception", "ValueError").And.NotContainInsertTexts("def", "abs"); if (is3X) { analysis = await GetAnalysisAsync("raise Exception from ", PythonVersions.LatestAvailable3X); - cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); result = cs.GetCompletions(analysis, new SourceLocation(1, 7)); result.Should().HaveInsertTexts("Exception", "ValueError").And.NotContainInsertTexts("def", "abs"); @@ -300,7 +305,7 @@ public async Task InRaise(bool is3X) { [TestMethod, Priority(0)] public async Task InExcept() { var analysis = await GetAnalysisAsync("try:\n pass\nexcept "); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 8)); result.Should().HaveInsertTexts("Exception", "ValueError").And.NotContainInsertTexts("def", "abs"); @@ -341,7 +346,7 @@ public async Task AfterDot() { x "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 3)); result.Should().HaveLabels("real", @"imag").And.NotContainLabels("abs"); @@ -368,7 +373,7 @@ public async Task AfterDot() { [TestMethod, Priority(0)] public async Task AfterAssign() { var analysis = await GetAnalysisAsync("x = x\ny = "); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(2, 4)); result.Should().HaveLabels("x", "abs"); @@ -388,7 +393,7 @@ def test_exception(self): self.assertRaises(TypeError). "; var analysis = await GetAnalysisAsync(code, is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(5, 38)); result.Should().HaveInsertTexts("exception"); @@ -400,7 +405,7 @@ public async Task WithWhitespaceAroundDot() { sys . version "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(2, 7)); result.Should().HaveLabels("argv"); @@ -409,7 +414,7 @@ sys . version [TestMethod, Priority(0)] public async Task MarkupKindValid() { var analysis = await GetAnalysisAsync("import sys\nsys.\n"); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(2, 5)); result.Completions?.Select(i => i.documentation?.kind).ExcludeDefault() @@ -426,7 +431,7 @@ from typing import NewType foo. "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(6, 5)); result.Should().HaveLabels("clear", "copy", "items", "keys", "update", "values"); @@ -443,7 +448,7 @@ def func(a: List[str]): pass "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(5, 7)); result.Should().HaveLabels("clear", "copy", "count", "index", "remove", "reverse"); @@ -463,7 +468,7 @@ def func(a: Dict[int, str]): pass "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(5, 7)); result.Should().HaveLabels("keys", "values"); @@ -493,7 +498,7 @@ def get(self) -> _T: y = boxedstr. "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(14, 14)); result.Should().HaveItem("get").Which.Should().HaveDocumentation("Box.get() -> int"); @@ -525,7 +530,7 @@ def get(self) -> _T: y = boxedstr. "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(14, 14)); result.Should().HaveLabels("append", "index"); @@ -554,7 +559,7 @@ def fob(self, x): def baz(self): pass "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var completionInD = cs.GetCompletions(analysis, new SourceLocation(3, 5)); var completionInOar = cs.GetCompletions(analysis, new SourceLocation(5, 9)); @@ -580,7 +585,7 @@ def abc(self): x.abc() "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var objectMemberNames = analysis.Document.Interpreter.GetBuiltinType(BuiltinTypeId.Object).GetMemberNames(); var completion = cs.GetCompletions(analysis, new SourceLocation(7, 1)); @@ -597,7 +602,7 @@ public async Task InFunctionDefinition(bool is3X) { var version = is3X ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X; var analysis = await GetAnalysisAsync("def f(a, b:int, c=2, d:float=None): pass", version); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 5)); result.Should().HaveNoCompletion(); @@ -631,7 +636,7 @@ public async Task InFunctionDefinition(bool is3X) { [TestMethod, Priority(0)] public async Task InFunctionDefinition_2X() { var analysis = await GetAnalysisAsync("@dec" + Environment.NewLine + "def f(): pass", PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 1)); result.Should().HaveLabels("any"); @@ -655,7 +660,7 @@ public async Task InFunctionDefinition_2X() { [TestMethod, Priority(0)] public async Task InFunctionDefinition_3X() { var analysis = await GetAnalysisAsync("@dec" + Environment.NewLine + "async def f(): pass", PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 1)); result.Should().HaveLabels("any"); @@ -683,7 +688,7 @@ public async Task InClassDefinition(bool is3x) { var version = is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X; var analysis = await GetAnalysisAsync("class C(object, parameter=MC): pass", version); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 8)); result.Should().HaveNoCompletion(); @@ -725,7 +730,7 @@ public async Task InClassDefinition(bool is3x) { [TestMethod, Priority(0)] public async Task InWithStatement() { var analysis = await GetAnalysisAsync("with x as y, z as w: pass"); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 6)); result.Should().HaveAnyCompletions(); @@ -779,7 +784,7 @@ public async Task ImportInPackage() { var analysis2 = await module2.GetAnalysisAsync(-1); var analysis3 = await module3.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis1, new SourceLocation(1, 16)); result.Should().OnlyHaveLabels("module2", "sub_package"); @@ -800,7 +805,7 @@ public async Task InImport() { "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(2, 7)); result.Should().HaveLabels("from", "import", "abs", "dir").And.NotContainLabels("abc"); @@ -886,7 +891,7 @@ def i(): pass def pass"; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 9)); result.Should().HaveNoCompletion(); @@ -904,7 +909,7 @@ def i(): pass public async Task NoCompletionInEllipsis(bool is2x) { const string code = "..."; var analysis = await GetAnalysisAsync(code, is2x ? PythonVersions.LatestAvailable2X : PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 4)); result.Should().HaveNoCompletion(); @@ -916,7 +921,7 @@ public async Task NoCompletionInEllipsis(bool is2x) { [DataTestMethod, Priority(0)] public async Task NoCompletionInString(bool is2x) { var analysis = await GetAnalysisAsync("\"str.\"", is2x ? PythonVersions.LatestAvailable2X : PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 6)); result.Should().HaveNoCompletion(); } @@ -924,7 +929,7 @@ public async Task NoCompletionInString(bool is2x) { [TestMethod, Priority(0)] public async Task NoCompletionInOpenString() { var analysis = await GetAnalysisAsync("'''."); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 5)); result.Should().HaveNoCompletion(); } @@ -935,7 +940,7 @@ public async Task NoCompletionInOpenString() { [DataTestMethod, Priority(0)] public async Task NoCompletionInFStringConstant(string openFString) { var analysis = await GetAnalysisAsync(openFString); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 5)); result.Should().HaveNoCompletion(); } @@ -943,7 +948,7 @@ public async Task NoCompletionInFStringConstant(string openFString) { [TestMethod, Priority(0)] public async Task NoCompletionBadImportExpression() { var analysis = await GetAnalysisAsync("import os,."); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); cs.GetCompletions(analysis, new SourceLocation(1, 12)); // Should not crash. } @@ -951,7 +956,7 @@ public async Task NoCompletionBadImportExpression() { public async Task NoCompletionInComment() { var analysis = await GetAnalysisAsync("x = 1 #str. more text"); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 12)); result.Should().HaveNoCompletion(); } @@ -965,7 +970,7 @@ import os os. "; var analysis = await GetAnalysisAsync(code, is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 4)); result.Should().HaveLabels("path", @"devnull", "SEEK_SET", @"curdir"); @@ -980,7 +985,7 @@ import os os.path. "; var analysis = await GetAnalysisAsync(code, is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 9)); result.Should().HaveLabels("split", @"getsize", @"islink", @"abspath"); @@ -990,7 +995,7 @@ import os public async Task FromDotInRoot() { const string code = "from ."; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 7)); result.Should().HaveNoCompletion(); @@ -1010,7 +1015,7 @@ public async Task FromDotInRootWithInitPy() { var analysis = await module1.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 7)); result.Should().OnlyHaveLabels("__dict__", "__file__", "__doc__", "__package__", "__debug__", "__name__", "__path__", "__spec__"); } @@ -1033,7 +1038,7 @@ public async Task FromDotInExplicitPackage() { await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await module.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 7)); result.Should().HaveLabels("module2", "sub_package", "answer"); @@ -1056,7 +1061,7 @@ public async Task FromPartialName() { await module.GetAnalysisAsync(-1); var analysis1 = await module1.GetAnalysisAsync(-1); var analysis2 = await module2.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis1, new SourceLocation(1, 8)); result.Should().HaveLabels("package").And.NotContainLabels("module2", "sub_package", "answer"); @@ -1078,12 +1083,31 @@ public async Task FromDotInImplicitPackage() { rdt.OpenDocument(module3, string.Empty); var analysis = await module.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 7)); result.Should().OnlyHaveLabels("module2", "sub_package"); } + [TestMethod, Priority(0)] + public async Task SubmoduleMember() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "__init__.py"), "from . import m1\nfrom . import m2\nx = 1"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "m1", "__init__.py"), string.Empty); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "m2", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "import package\npackage."); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + + var result = cs.GetCompletions(analysis, new SourceLocation(2, 9)); + result.Should().HaveLabels("m1", "m2", "x"); + } + [DataRow(false)] [DataRow(true)] [DataTestMethod, Priority(0)] @@ -1093,7 +1117,7 @@ from os.path import exists as EX E "; var analysis = await GetAnalysisAsync(code, is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 2)); result.Should().HaveLabels("EX"); @@ -1106,7 +1130,7 @@ from os.path import exists as EX public async Task NoDuplicateMembers() { const string code = @"import sy"; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, 10)); result.Completions.Count(c => c.label.EqualsOrdinal(@"sys")).Should().Be(1); @@ -1123,7 +1147,7 @@ class A: ... a. "; var analysis = await GetAnalysisAsync(code, is3x ? PythonVersions.LatestAvailable3X : PythonVersions.LatestAvailable2X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var extraMembers = new[] { "mro", "__dict__", @"__weakref__" }; var result = cs.GetCompletions(analysis, new SourceLocation(4, 3)); if (is3x) { @@ -1150,7 +1174,7 @@ def main(req: func.HttpRequest) -> func.HttpResponse: $"'azure.functions' package is not installed for Python {ver}, see https://github.com/Microsoft/python-language-server/issues/462"); } - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(5, 23)); result.Should().HaveLabels("get"); result.Completions.First(x => x.label == "get").Should().HaveDocumentation("dict.get*"); @@ -1162,7 +1186,7 @@ public async Task InForEnumeration() { for a, b in x: "); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(3, 4)); result.Should().HaveLabels("a", "b"); } @@ -1175,7 +1199,7 @@ public async Task NoCompletionForCurrentModuleName(bool empty) { var code = empty ? string.Empty : $"{Path.GetFileNameWithoutExtension(modulePath)}."; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X, null, modulePath); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(1, code.Length + 1)); result.Should().NotContainLabels(analysis.Document.Name); } @@ -1193,7 +1217,7 @@ import sys var analysis = await GetDocumentAnalysisAsync(doc); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var completions = cs.GetCompletions(analysis, new SourceLocation(3, 5)); completions.Should().HaveLabels("argv", "path", "exit"); } @@ -1205,7 +1229,7 @@ def func(): aaa = 1 a"; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(4, 2)); result.Completions.Select(c => c.label).Should().NotContain("aaa"); @@ -1224,7 +1248,7 @@ def func(self): A(). "; var analysis = await GetAnalysisAsync(code); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var result = cs.GetCompletions(analysis, new SourceLocation(7, 14)); result.Completions.Select(c => c.label).Should().Contain("__x").And.NotContain("_A__x"); @@ -1250,7 +1274,7 @@ def test(x: Foo = func()): x. "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(13, 7)); comps.Should().HaveLabels("name", "z"); } @@ -1261,7 +1285,7 @@ public async Task AddBrackets() { var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); ServerSettings.completion.addBrackets = true; - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(1, 5)); var print = comps.Completions.FirstOrDefault(x => x.label == "print"); @@ -1293,7 +1317,7 @@ def method2(self): "; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(11, 21)); var names = comps.Completions.Select(c => c.label); @@ -1303,5 +1327,189 @@ def method2(self): names = comps.Completions.Select(c => c.label); names.Should().NotContain(new[] { "x1", "x2", "method1", "method2", "B" }); } + + [TestMethod, Priority(0)] + public async Task FromImportPackageNoInitPy() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("top", "sub1", "__init__.py"), string.Empty); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "from top import s"); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var comps = cs.GetCompletions(analysis, new SourceLocation(1, 18)); + var names = comps.Completions.Select(c => c.label); + names.Should().Contain(new[] { "sub1" }); + } + + [TestMethod, Priority(0)] + public async Task InFunctionParameters() { + const string code = @" +class A: + def method(self, content=1): + return content + +class B: + def __init__(self, ctorParam = 2): + pass + + def method(self): + a = A() + a.method() + +a = A() +a.method() + +b = B() +"; + var analysis = await GetAnalysisAsync(code); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + + var comps = cs.GetCompletions(analysis, new SourceLocation(12, 18)); + comps.Should().HaveLabels("content="); + + comps = cs.GetCompletions(analysis, new SourceLocation(15, 10)); + comps.Should().HaveLabels("content="); + + comps = cs.GetCompletions(analysis, new SourceLocation(17, 7)); + comps.Should().HaveLabels("ctorParam="); + } + + [TestMethod] + public async Task PositionalOnly() { + const string code = @" +def func(a, b, /, c, d, *, e, f): + pass + +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var result = cs.GetCompletions(analysis, new SourceLocation(4, 1)); + + result.Should().HaveItem("func") + .Which.Should().HaveDocumentation("func(a, b, /, c, d, *, e, f)"); + } + + [TestMethod] + public async Task PositionalOnlyOverride() { + const string code = @" +class A: + def foo(self, x, y, /, *args): + pass + +class B(A): + def +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var result = cs.GetCompletions(analysis, new SourceLocation(7, 9)); + + result.Should().HaveItem("foo") + .Which.Should().HaveInsertText($"foo(self, x, y, /, *args):{Environment.NewLine} return super().foo(x, y, *args)") + .And.HaveInsertTextFormat(InsertTextFormat.PlainText); + } + + [TestMethod] + public async Task OnlyOneNone() { + var analysis = await GetAnalysisAsync("", PythonVersions.Required_Python38X); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var result = cs.GetCompletions(analysis, new SourceLocation(1, 1)); + + result.Completions.Where(item => item.insertText == "None").Should().HaveCount(1); + } + + + + [TestMethod] + public async Task SingleInheritanceSuperCheckCompletion() { + const string code = @" +class A: + def base_func(self): + return 1234 + +class B(A): + def foo(self): + x = super() + x. +"; + + var analysis = await GetAnalysisAsync(code); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var result = cs.GetCompletions(analysis, new SourceLocation(9, 11)); + + result.Completions.Where(item => item.insertText == "base_func").Should().HaveCount(1); + } + + + [TestMethod] + public async Task SingleInheritanceSuperCheckCompletionNoDuplicates() { + const string code = @" +class A: + def foo(self): + return 1234 + +class B(A): + def foo(self): + return 4321 + +class C(B): + def foo(self): + super(). +"; + var analysis = await GetAnalysisAsync(code); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + var result = cs.GetCompletions(analysis, new SourceLocation(12, 17)); + + result.Completions.GroupBy(item => item.insertText).Any(g => g.Count() > 1).Should().BeFalse(); + } + + + [TestMethod, Priority(0)] + public async Task ImportDotMembers() { + var appUri = TestData.GetTestSpecificUri("app.py"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "__init__.py"), "badvar1 = 3.141"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "m1", "__init__.py"), "badvar2 = 123"); + await TestData.CreateTestSpecificFileAsync(Path.Combine("package", "m2", "__init__.py"), "badvar3 = 'str'"); + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + var doc = rdt.OpenDocument(appUri, "import package."); + + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await doc.GetAnalysisAsync(Timeout.Infinite); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + + var result = cs.GetCompletions(analysis, new SourceLocation(1, 16)); + result.Should().OnlyHaveLabels("m1", "m2"); + } + + [TestMethod, Priority(0)] + public async Task ClassPrivateMembers() { + const string code = @" +class Op: + __EQ = '0' + __NOT_EQ = '1' + __OP_LIST = [__EQ] + + def print_EQ1(self): + self. + + @classmethod + def print_EQ2(cls): + cls. +"; + var analysis = await GetAnalysisAsync(code); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); + + var result = cs.GetCompletions(analysis, new SourceLocation(8, 14)); + result.Should().HaveLabels("__EQ", "__NOT_EQ", "__OP_LIST"); + + result = cs.GetCompletions(analysis, new SourceLocation(12, 13)); + result.Should().HaveLabels("__EQ", "__NOT_EQ", "__OP_LIST"); + } } } diff --git a/src/LanguageServer/Test/DiagnosticsTests.cs b/src/LanguageServer/Test/DiagnosticsTests.cs index d105bcbc7..99b510448 100644 --- a/src/LanguageServer/Test/DiagnosticsTests.cs +++ b/src/LanguageServer/Test/DiagnosticsTests.cs @@ -148,7 +148,9 @@ public async Task CloseDocument() { clientApp.When(x => x.NotifyWithParameterObjectAsync("textDocument/publishDiagnostics", Arg.Any())) .Do(x => reported.Add(x.Args()[1] as PublishDiagnosticsParams)); - doc.Dispose(); + var rdt = Services.GetService(); + rdt.CloseDocument(doc.Uri); + ds.Diagnostics.TryGetValue(doc.Uri, out _).Should().BeFalse(); reported.Count.Should().Be(1); reported[0].uri.Should().Be(doc.Uri); diff --git a/src/LanguageServer/Test/DocstringConverterTests.cs b/src/LanguageServer/Test/DocstringConverterTests.cs index f2c85516d..a57392d04 100644 --- a/src/LanguageServer/Test/DocstringConverterTests.cs +++ b/src/LanguageServer/Test/DocstringConverterTests.cs @@ -501,6 +501,13 @@ public void AsteriskList() { docstring.Should().ConvertToMarkdown(markdown); } + [TestMethod, Priority(0)] + public void SquareBrackets() { + var docstring = @"Optional[List[str]]"; + var markdown = @"Optional\[List\[str\]\]"; + docstring.Should().ConvertToMarkdown(markdown); + } + [DataRow(null)] [DataRow("")] [DataRow(" ")] diff --git a/src/LanguageServer/Test/DocumentHighlightTests.cs b/src/LanguageServer/Test/DocumentHighlightTests.cs new file mode 100644 index 000000000..4781b9db8 --- /dev/null +++ b/src/LanguageServer/Test/DocumentHighlightTests.cs @@ -0,0 +1,85 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.IO; +using System.Threading.Tasks; +using FluentAssertions; +using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Analysis.Documents; +using Microsoft.Python.Core.Text; +using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.LanguageServer.Sources; +using Microsoft.Python.LanguageServer.Tests.FluentAssertions; +using Microsoft.Python.Parsing.Tests; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using TestUtilities; + +namespace Microsoft.Python.LanguageServer.Tests { + [TestClass] + public class DocumentHighlightTests : LanguageServerTestBase { + public TestContext TestContext { get; set; } + + [TestInitialize] + public void TestInitialize() + => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + + [TestCleanup] + public void Cleanup() => TestEnvironmentImpl.TestCleanup(); + + + [TestMethod, Priority(0)] + public async Task HighlightBasic() { + const string code = @" +x = 1 + +def func(x): + return x + +y = func(x) +x = 2 +"; + var analysis = await GetAnalysisAsync(code); + var dhs = new DocumentHighlightSource(Services); + + // Test global scope + var highlights1 = await dhs.DocumentHighlightAsync(analysis.Document.Uri, new SourceLocation(8, 1)); + + highlights1.Should().HaveCount(3); + highlights1[0].range.Should().Be(1, 0, 1, 1); + highlights1[0].kind.Should().Be(DocumentHighlightKind.Write); + highlights1[1].range.Should().Be(6, 9, 6, 10); + highlights1[1].kind.Should().Be(DocumentHighlightKind.Read); + highlights1[2].range.Should().Be(7, 0, 7, 1); + + // Test local scope in func() + var highlights2 = await dhs.DocumentHighlightAsync(analysis.Document.Uri, new SourceLocation(4, 10)); + + highlights2.Should().HaveCount(2); + highlights2[0].range.Should().Be(3, 9, 3, 10); + highlights2[0].kind.Should().Be(DocumentHighlightKind.Write); + highlights2[1].range.Should().Be(4, 11, 4, 12); + highlights2[1].kind.Should().Be(DocumentHighlightKind.Read); + } + + [TestMethod, Priority(0)] + public async Task HighlightEmptyDocument() { + await GetAnalysisAsync(string.Empty); + var dhs = new DocumentHighlightSource(Services); + var references = await dhs.DocumentHighlightAsync(null, new SourceLocation(1, 1)); + references.Should().BeEmpty(); + } + } +} diff --git a/src/LanguageServer/Test/GoToDefinitionTests.cs b/src/LanguageServer/Test/GoToDefinitionTests.cs index 082ea06c2..3de1b1424 100644 --- a/src/LanguageServer/Test/GoToDefinitionTests.cs +++ b/src/LanguageServer/Test/GoToDefinitionTests.cs @@ -165,6 +165,83 @@ def hello(self): reference.range.Should().Be(2, 4, 2, 5); } + [TestMethod, Priority(0)] + public async Task GotoDefinitionFromParentOtherModuleOnSuper() { + var otherModPath = TestData.GetTestSpecificUri("other.py"); + var testModPath = TestData.GetTestSpecificUri("test.py"); + const string otherModCode = @" +class C: + v: int + def test(self): + pass +"; + const string testModCode = @" +from other import C + +class D(C): + def hello(self): + super().test(); + super().v +"; + + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + + rdt.OpenDocument(otherModPath, otherModCode); + var testMod = rdt.OpenDocument(testModPath, testModCode); + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await testMod.GetAnalysisAsync(); + var ds = new DefinitionSource(Services); + + var reference = ds.FindDefinition(analysis, new SourceLocation(6, 17), out _); + reference.Should().NotBeNull(); + reference.uri.AbsolutePath.Should().Contain("other.py"); + reference.range.Should().Be(3, 8, 3, 12); + + reference = ds.FindDefinition(analysis, new SourceLocation(7, 17), out _); + reference.Should().NotBeNull(); + reference.uri.AbsolutePath.Should().Contain("other.py"); + reference.range.Should().Be(2, 4, 2, 5); + } + + [TestMethod, Priority(0)] + [Ignore("Todo: super() multiple Inheritance support")] + public async Task MultipleInheritanceSuperShouldWalkDecendants() { + var testModPath = TestData.GetTestSpecificUri("test.py"); + const string code = @" + class GrandParent: + def dowork(self): + return 1 + + class Dad(GrandParent): + def dowork(self): + return super().dowork() + + class Mom(): + def dowork(self): + return 2 + + class Child(Dad, Mom): + def child_func(self): + pass + + "; + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + + var testMod = rdt.OpenDocument(testModPath, code); + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await testMod.GetAnalysisAsync(); + var ds = new DefinitionSource(Services); + + // Goto on Dad's super().dowork() should jump to Mom's + var reference = ds.FindDefinition(analysis, new SourceLocation(9, 33), out _); + reference.Should().NotBeNull(); + reference.uri.AbsolutePath.Should().Contain("test.py"); + reference.range.Should().Be(11, 16, 11, 23); + } + + [TestMethod, Priority(0)] public async Task GotoModuleSource() { const string code = @" @@ -572,5 +649,90 @@ import datetime reference.uri.AbsolutePath.Should().Contain("datetime.py"); reference.uri.AbsolutePath.Should().NotContain("pyi"); } + + [TestMethod, Priority(0)] + public async Task GotoDefinitionFromMultiImport() { + var otherModPath = TestData.GetTestSpecificUri("other.py"); + var testModPath = TestData.GetTestSpecificUri("test.py"); + const string otherModCode = @" +def a(): ... +def b(): ... +def c(): ... +"; + const string testModCode = @" +from other import a, b, c +"; + await CreateServicesAsync(PythonVersions.LatestAvailable3X); + var rdt = Services.GetService(); + + rdt.OpenDocument(otherModPath, otherModCode); + var testMod = rdt.OpenDocument(testModPath, testModCode); + await Services.GetService().WaitForCompleteAnalysisAsync(); + var analysis = await testMod.GetAnalysisAsync(); + var ds = new DefinitionSource(Services); + + var reference = ds.FindDefinition(analysis, new SourceLocation(2, 19), out _); + reference.Should().NotBeNull(); + reference.uri.AbsolutePath.Should().Contain("other.py"); + reference.range.Should().Be(1, 4, 1, 5); + + reference = ds.FindDefinition(analysis, new SourceLocation(2, 22), out _); + reference.Should().NotBeNull(); + reference.uri.AbsolutePath.Should().Contain("other.py"); + reference.range.Should().Be(2, 4, 2, 5); + + reference = ds.FindDefinition(analysis, new SourceLocation(2, 25), out _); + reference.Should().NotBeNull(); + reference.uri.AbsolutePath.Should().Contain("other.py"); + reference.range.Should().Be(3, 4, 3, 5); + } + + [TestMethod, Priority(0)] + public async Task UnknownType() { + const string code = @" +A +"; + var analysis = await GetAnalysisAsync(code); + var ds = new DefinitionSource(Services); + var reference = ds.FindDefinition(analysis, new SourceLocation(2, 1), out _); + reference.Should().BeNull(); + } + + [TestMethod, Priority(0)] + public async Task UnknownImportedType() { + const string code = @" +from nonexistent import some +"; + var analysis = await GetAnalysisAsync(code); + var ds = new DefinitionSource(Services); + var reference = ds.FindDefinition(analysis, new SourceLocation(2, 26), out _); + reference.Should().BeNull(); + } + + [TestMethod, Priority(0)] + public async Task LocalParameter() { + const string code = @" +def func(a, b): + return a+b +"; + var analysis = await GetAnalysisAsync(code); + var ds = new DefinitionSource(Services); + var reference = ds.FindDefinition(analysis, new SourceLocation(3, 12), out _); + reference.range.Should().Be(1, 9, 1, 10); + } + + [TestMethod, Priority(0)] + public async Task CompiledCode() { + const string code = @" +import os +os.mkdir() +"; + var analysis = await GetAnalysisAsync(code); + var ds = new DefinitionSource(Services); + var reference = ds.FindDefinition(analysis, new SourceLocation(3, 6), out _); + reference.range.start.line.Should().NotBe(1); + reference.range.end.line.Should().NotBe(1); + reference.uri.AbsolutePath.Should().Contain(".pyi"); + } } } diff --git a/src/LanguageServer/Test/HoverTests.cs b/src/LanguageServer/Test/HoverTests.cs index 12b7732db..9377e5cc0 100644 --- a/src/LanguageServer/Test/HoverTests.cs +++ b/src/LanguageServer/Test/HoverTests.cs @@ -189,8 +189,8 @@ from os.path import join as JOIN var name = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? @"ntpath" : @"posixpath"; AssertHover(hs, analysis, new SourceLocation(2, 10), $"module {name}*", new SourceSpan(2, 9, 2, 13)); - AssertHover(hs, analysis, new SourceLocation(2, 22), @"join(path: str, paths: str) -> str", new SourceSpan(2, 21, 2, 25)); - AssertHover(hs, analysis, new SourceLocation(2, 30), @"join(path: str, paths: str) -> str", new SourceSpan(2, 29, 2, 33)); + AssertHover(hs, analysis, new SourceLocation(2, 22), @"join(path: str, *paths: str) -> str", new SourceSpan(2, 21, 2, 25)); + AssertHover(hs, analysis, new SourceLocation(2, 30), @"join(path: str, *paths: str) -> str", new SourceSpan(2, 29, 2, 33)); } [TestMethod, Priority(0)] @@ -229,7 +229,7 @@ public async Task AssignmentExpressions() { const string code = @" (a := 1) "; - var analysis = await GetAnalysisAsync(code); + var analysis = await GetAnalysisAsync(code, PythonVersions.Required_Python38X); var hs = new HoverSource(new PlainTextDocumentationSource()); AssertHover(hs, analysis, new SourceLocation(2, 2), @"a: int", new SourceSpan(2, 2, 2, 3)); } @@ -284,6 +284,18 @@ class A: AssertHover(hs, analysis, new SourceLocation(4, 7), @"x: int", new SourceSpan(4, 7, 4, 8)); } + [TestMethod, Priority(0)] + public async Task CompiledCode() { + const string code = @" +import os +os.mkdir() +"; + var analysis = await GetAnalysisAsync(code); + var hs = new HoverSource(new PlainTextDocumentationSource()); + var hover = hs.GetHover(analysis, new SourceLocation(3, 6)); + hover.contents.value.Should().Contain("Create a directory"); + } + private static void AssertNoHover(HoverSource hs, IDocumentAnalysis analysis, SourceLocation position) { var hover = hs.GetHover(analysis, position); hover.Should().BeNull(); diff --git a/src/LanguageServer/Test/ImportsTests.cs b/src/LanguageServer/Test/ImportsTests.cs index fd3b4824b..b114e05ff 100644 --- a/src/LanguageServer/Test/ImportsTests.cs +++ b/src/LanguageServer/Test/ImportsTests.cs @@ -71,7 +71,7 @@ import projectB.foo.baz var doc = rdt.OpenDocument(new Uri(appPath), appCode, appPath); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(7, 10)); comps.Should().HaveLabels("foo"); @@ -117,7 +117,7 @@ from projectB.foo import baz var doc = rdt.OpenDocument(new Uri(appPath), appCode, appPath); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(8, 10)); comps.Should().HaveLabels("foo"); @@ -154,7 +154,7 @@ public async Task SysModuleChain() { await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await doc1.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(2, 5)); comps.Should().HaveLabels("VALUE"); } @@ -176,7 +176,7 @@ await TestData.CreateTestSpecificFileAsync("module2.py", @"import sys await Services.GetService().WaitForCompleteAnalysisAsync(); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(2, 5)); comps.Should().HaveLabels("VALUE"); } @@ -202,7 +202,7 @@ public async Task UncSearchPaths() { var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(1, 21)); comps.Should().HaveLabels("module1", "module2"); @@ -257,7 +257,7 @@ def method2(): await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(2, 6)); comps.Should().HaveLabels("A").And.NotContainLabels("B"); @@ -296,7 +296,7 @@ import package.sub_package.module2 var doc = rdt.OpenDocument(new Uri(appPath), appCode); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(5, 9)); comps.Should().OnlyHaveLabels("sub_package"); @@ -333,7 +333,7 @@ import package.module.submodule as submodule var doc = rdt.OpenDocument(appUri, appCode); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(4, 8)); comps.Should().HaveLabels("Y").And.NotContainLabels("X"); @@ -364,7 +364,7 @@ from package.module import submodule var doc = rdt.OpenDocument(appUri, appCode); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(4, 8)); comps.Should().HaveLabels("Y").And.NotContainLabels("X"); @@ -395,7 +395,7 @@ from .sub_package.module import submodule var doc = rdt.OpenDocument(new Uri(appPath), appCode); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(4, 8)); comps.Should().HaveLabels("Y").And.NotContainLabels("X"); @@ -464,7 +464,7 @@ from module3 import A3 await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await app.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(10, 4)); comps.Should().HaveLabels("M2"); @@ -478,7 +478,7 @@ from module3 import A3 [TestMethod, Priority(0)] public async Task TypingModule() { var analysis = await GetAnalysisAsync(@"from typing import "); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(1, 20)); comps.Should().HaveLabels("TypeVar", "List", "Dict", "Union"); } @@ -503,7 +503,7 @@ public async Task RelativeImportsFromParent() { await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await module2.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(2, 9)); comps.Should().HaveLabels("X"); } @@ -528,7 +528,7 @@ public async Task FromImport_ModuleAffectsPackage(string appCodeImport) { var doc = rdt.OpenDocument(new Uri(appPath), appCode1); var analysis = await doc.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(2, 13)); comps.Should().OnlyHaveLabels("module"); @@ -581,7 +581,7 @@ from module1 import * await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await app.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(4, 5)); comps.Should().HaveLabels("foo"); @@ -644,7 +644,7 @@ from module1 import * await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await app.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(4, 5)); comps.Should().HaveLabels("foo"); @@ -707,7 +707,7 @@ from module1 import * await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await app.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(4, 5)); comps.Should().HaveLabels("foo"); @@ -751,7 +751,7 @@ from module1 import _B as B await analyzer.WaitForCompleteAnalysisAsync(); var analysis = await app.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(5, 5)); comps.Should().HaveLabels("foo"); @@ -790,7 +790,7 @@ public async Task Python2XRelativeImportInRoot() { var analysis = await app.GetAnalysisAsync(-1); var analysisInPackage = await appInPackage.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(2, 8)); comps.Should().HaveLabels("X"); @@ -824,7 +824,7 @@ import module2 var app = rdt.OpenDocument(appUri, appContent); var analysis = await app.GetAnalysisAsync(-1); - var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion); + var cs = new CompletionSource(new PlainTextDocumentationSource(), ServerSettings.completion, Services); var comps = cs.GetCompletions(analysis, new SourceLocation(3, 9)); comps.Should().HaveLabels("X"); diff --git a/src/LanguageServer/Test/LineFormatterTests.cs b/src/LanguageServer/Test/LineFormatterTests.cs index fbddced98..86c2c95e7 100644 --- a/src/LanguageServer/Test/LineFormatterTests.cs +++ b/src/LanguageServer/Test/LineFormatterTests.cs @@ -428,6 +428,11 @@ public void FStrings(string code, string expected) { AssertSingleLineFormat(code, expected); } + [TestMethod, Priority(0)] + public void PositionalMarker() { + AssertSingleLineFormat("def foo(x,y,/,z,*args,**kwargs)", "def foo(x, y, /, z, *args, **kwargs)"); + } + [TestMethod, Priority(0)] public void GrammarFile() { var src = TestData.GetPath("TestData", "Formatting", "pythonGrammar.py"); diff --git a/src/LanguageServer/Test/Microsoft.Python.LanguageServer.Tests.csproj b/src/LanguageServer/Test/Microsoft.Python.LanguageServer.Tests.csproj index dd43c522b..3b8375161 100644 --- a/src/LanguageServer/Test/Microsoft.Python.LanguageServer.Tests.csproj +++ b/src/LanguageServer/Test/Microsoft.Python.LanguageServer.Tests.csproj @@ -1,6 +1,6 @@  - netcoreapp3.0 + netcoreapp3.1 Microsoft.Python.LanguageServer.Tests Microsoft.Python.LanguageServer.Tests @@ -23,15 +23,15 @@ - - - - + + + + all runtime; build; native; contentfiles; analyzers - + diff --git a/src/LanguageServer/Test/MissingImportCodeActionTests.cs b/src/LanguageServer/Test/MissingImportCodeActionTests.cs new file mode 100644 index 000000000..69759b049 --- /dev/null +++ b/src/LanguageServer/Test/MissingImportCodeActionTests.cs @@ -0,0 +1,542 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using FluentAssertions; +using Microsoft.Python.Analysis; +using Microsoft.Python.Analysis.Analyzer; +using Microsoft.Python.Core.Idle; +using Microsoft.Python.Core.IO; +using Microsoft.Python.Core.Services; +using Microsoft.Python.Core.Text; +using Microsoft.Python.LanguageServer.CodeActions; +using Microsoft.Python.LanguageServer.Diagnostics; +using Microsoft.Python.LanguageServer.Indexing; +using Microsoft.Python.LanguageServer.Protocol; +using Microsoft.Python.LanguageServer.Sources; +using Microsoft.Python.LanguageServer.Tests.FluentAssertions; +using Microsoft.Python.Parsing.Ast; +using Microsoft.Python.Parsing.Tests; +using Microsoft.Python.UnitTests.Core; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using TestUtilities; + +namespace Microsoft.Python.LanguageServer.Tests { + [TestClass] + public class MissingImportCodeActionTests : LanguageServerTestBase { + public TestContext TestContext { get; set; } + public CancellationToken CancellationToken => TestContext.CancellationTokenSource.Token; + + [TestInitialize] + public void TestInitialize() + => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + + [TestCleanup] + public void Cleanup() => TestEnvironmentImpl.TestCleanup(); + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task Missing() { + MarkupUtils.GetSpan(@"[|missingModule|]", out var code, out var span); + + var analysis = await GetAnalysisAsync(code); + var diagnostics = GetDiagnostics(analysis, span.ToSourceSpan(analysis.Ast), MissingImportCodeActionProvider.Instance.FixableDiagnostics); + diagnostics.Should().NotBeEmpty(); + + var codeActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync(analysis, CodeActionSettings.Default, diagnostics, CancellationToken); + codeActions.Should().BeEmpty(); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task TopModule() { + const string markup = @"{|insertionSpan:|}{|diagnostic:ntpath|}"; + + var (analysis, codeActions, insertionSpan) = await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + + var codeAction = codeActions.Single(); + var newText = "import ntpath" + Environment.NewLine + Environment.NewLine; + TestCodeAction(analysis.Document.Uri, codeAction, title: "import ntpath", insertionSpan, newText); + } + + [TestMethod, Priority(0), Ignore, Timeout(AnalysisTimeoutInMS)] + public async Task TopModuleFromFunctionInsertTop() { + const string markup = @"{|insertionSpan:|}def TestMethod(): + {|diagnostic:ntpath|}"; + + var (analysis, codeActions, insertionSpan) = await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + + codeActions.Should().HaveCount(2); + + var codeAction = codeActions.First(); + var newText = "import ntpath" + Environment.NewLine + Environment.NewLine; + TestCodeAction(analysis.Document.Uri, codeAction, title: "import ntpath", insertionSpan, newText); + } + + [TestMethod, Priority(0), Ignore, Timeout(AnalysisTimeoutInMS)] + public async Task TopModuleLocally() { + const string markup = @"def TestMethod(): +{|insertionSpan:|} {|diagnostic:ntpath|}"; + + var (analysis, codeActions, insertionSpan) = await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + + codeActions.Should().HaveCount(2); + + var codeAction = codeActions[1]; + var newText = " import ntpath" + Environment.NewLine + Environment.NewLine; + TestCodeAction(analysis.Document.Uri, codeAction, title: string.Format(Resources.ImportLocally, "import ntpath"), insertionSpan, newText); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SubModule() { + await TestCodeActionAsync( + @"{|insertionSpan:|}{|diagnostic:util|}", + title: "from ctypes import util", + newText: "from ctypes import util" + Environment.NewLine + Environment.NewLine); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SubModuleUpdate() { + await TestCodeActionAsync( + @"{|insertionSpan:from ctypes import util|} +{|diagnostic:test|}", + title: "from ctypes import test, util", + newText: "from ctypes import test, util"); + } + + [TestMethod, Priority(0), Ignore, Timeout(AnalysisTimeoutInMS)] + public async Task SubModuleUpdateLocally() { + await TestCodeActionAsync( + @"def TestMethod(): + {|insertionSpan:from ctypes import util|} + {|diagnostic:test|}", + title: string.Format(Resources.ImportLocally, "from ctypes import test, util"), + newText: "from ctypes import test, util"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SubModuleFromFunctionInsertTop() { + await TestCodeActionAsync( + @"{|insertionSpan:|}def TestMethod(): + from ctypes import util + {|diagnostic:test|}", + title: "from ctypes import test", + newText: "from ctypes import test" + Environment.NewLine + Environment.NewLine); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task AfterExistingImport() { + await TestCodeActionAsync( + @"from os import path +{|insertionSpan:|} +{|diagnostic:util|}", + title: "from ctypes import util", + newText: "from ctypes import util" + Environment.NewLine); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task ReplaceExistingImport() { + await TestCodeActionAsync( + @"from os import path +{|insertionSpan:from ctypes import test|} +import socket + +{|diagnostic:util|}", + title: "from ctypes import test, util", + newText: "from ctypes import test, util"); + } + + [TestMethod, Priority(0), Ignore, Timeout(AnalysisTimeoutInMS)] + public async Task AfterExistingImportLocally() { + await TestCodeActionAsync( + @"def TestMethod(): + from os import path +{|insertionSpan:|} + {|diagnostic:util|}", + title: string.Format(Resources.ImportLocally, "from ctypes import util"), + newText: " from ctypes import util" + Environment.NewLine); + } + + [TestMethod, Priority(0), Ignore, Timeout(AnalysisTimeoutInMS)] + public async Task ReplaceExistingImportLocally() { + await TestCodeActionAsync( + @"def TestMethod(): + from os import path + {|insertionSpan:from ctypes import test|} + import socket + + {|diagnostic:util|}", + title: string.Format(Resources.ImportLocally, "from ctypes import test, util"), + newText: "from ctypes import test, util"); + } + + [TestMethod, Priority(0), Ignore, Timeout(AnalysisTimeoutInMS)] + public async Task CodeActionOrdering() { + MarkupUtils.GetSpan(@"def TestMethod(): + [|test|]", out var code, out var span); + + var analysis = await GetAnalysisAsync(code); + var diagnostics = GetDiagnostics(analysis, span.ToSourceSpan(analysis.Ast), MissingImportCodeActionProvider.Instance.FixableDiagnostics); + diagnostics.Should().NotBeEmpty(); + + var codeActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync(analysis, CodeActionSettings.Default, diagnostics, CancellationToken); + + var list = codeActions.Select(c => c.title).ToList(); + var zipList = Enumerable.Range(0, list.Count).Zip(list); + + var locallyImportedPrefix = Resources.ImportLocally.Substring(0, Resources.ImportLocally.IndexOf("'")); + var maxIndexOfTopAddImports = zipList.Where(t => !t.Second.StartsWith(locallyImportedPrefix)).Max(t => t.First); + var minIndexOfLocalAddImports = zipList.Where(t => t.Second.StartsWith(locallyImportedPrefix)).Min(t => t.First); + + maxIndexOfTopAddImports.Should().BeLessThan(minIndexOfLocalAddImports); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task PreserveComment() { + await TestCodeActionAsync( + @"{|insertionSpan:from os import pathconf|} # test + +{|diagnostic:path|}", + title: "from os import path, pathconf", + newText: "from os import path, pathconf"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task MemberSymbol() { + await TestCodeActionAsync( + @"from os import path +{|insertionSpan:|} +{|diagnostic:socket|}", + title: "from socket import socket", + newText: "from socket import socket" + Environment.NewLine); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task NoMemberSymbol() { + var markup = @"{|insertionSpan:|}{|diagnostic:socket|}"; + + var (analysis, codeActions, insertionSpan) = await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + + codeActions.Select(c => c.title).Should().NotContain("from socket import socket"); + + var title = "import socket"; + var codeAction = codeActions.Single(c => c.title == title); + var newText = "import socket" + Environment.NewLine + Environment.NewLine; + TestCodeAction(analysis.Document.Uri, codeAction, title, insertionSpan, newText); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SymbolOrdering() { + var markup = @"from os import path +{|insertionSpan:|} +{|diagnostic:socket|}"; + + var (analysis, codeActions, insertionSpan) = await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + + var list = codeActions.Select(c => c.title).ToList(); + var zipList = Enumerable.Range(0, list.Count).Zip(list); + + var maxIndexOfPublicSymbol = zipList.Where(t => !t.Second.StartsWith("from _")).Max(t => t.First); + var minIndexOfPrivateSymbol = zipList.Where(t => t.Second.StartsWith("from _")).Min(t => t.First); + + minIndexOfPrivateSymbol.Should().BeLessThan(maxIndexOfPublicSymbol); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS*2)] + public async Task SymbolOrdering2() { + var markup = @"from os import path +{|insertionSpan:|} +{|diagnostic:join|}"; + + var (analysis, codeActions, insertionSpan) = await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics, enableIndexManager: true); + + var list = codeActions.Select(c => c.title).ToList(); + var zipList = Enumerable.Range(0, list.Count).Zip(list); + + var sourceDeclIndex = zipList.First(t => t.Second == "from posixpath import join").First; + var importedMemberIndex = zipList.First(t => t.Second == "from os.path import join").First; + var restIndex = zipList.First(t => t.Second == "from ntpath import join").First; + + sourceDeclIndex.Should().BeLessThan(importedMemberIndex); + importedMemberIndex.Should().BeLessThan(restIndex); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SymbolOrdering3() { + var markup = @"{|insertionSpan:|}{|diagnostic:pd|}"; + + MarkupUtils.GetNamedSpans(markup, out var code, out var spans); + + // get main analysis and add mock modules + var analysis = await GetAnalysisAsync(code); + + await GetAnalysisAsync("", analysis.ExpressionEvaluator.Services, modulePath: TestData.GetTestSpecificPath("pandas.py")); + await GetAnalysisAsync("", analysis.ExpressionEvaluator.Services, modulePath: TestData.GetTestSpecificPath("pd.py")); + + // calculate actions + var diagnosticSpan = spans["diagnostic"].First().ToSourceSpan(analysis.Ast); + var diagnostics = GetDiagnostics(analysis, diagnosticSpan, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + var codeActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync(analysis, CodeActionSettings.Default, diagnostics, CancellationToken); + + var list = codeActions.Select(c => c.title).ToList(); + var zipList = Enumerable.Range(0, list.Count).Zip(list); + + var pandasIndex = zipList.First(t => t.Second == "import pandas as pd").First; + var pdIndex = zipList.First(t => t.Second == "import pd").First; + + pandasIndex.Should().BeLessThan(pdIndex); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + [Ignore("loops")] + public async Task ModuleNotReachableFromUserDocument() { + await TestCodeActionAsync( + @"{|insertionSpan:|}{|diagnostic:path|}", + title: "from os import path", + newText: "from os import path" + Environment.NewLine + Environment.NewLine, + enableIndexManager: true); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SuggestAbbreviationForKnownModule() { + await TestCodeActionAsync( + @"{|insertionSpan:|}{|diagnostic:pandas|}", + title: "import pandas as pd", + newText: "import pandas as pd" + Environment.NewLine + Environment.NewLine, + abbreviation: "pd", + relativePaths: "pandas.py"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SuggestAbbreviationForKnownModule2() { + await TestCodeActionAsync( + @"{|insertionSpan:|}{|diagnostic:pyplot|}", + title: "from matplotlib import pyplot as plt", + newText: "from matplotlib import pyplot as plt" + Environment.NewLine + Environment.NewLine, + abbreviation: "plt", + relativePaths: @"matplotlib\pyplot.py"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SuggestAbbreviationForKnownModule3() { + var markup = @" +{|insertionSpan:from matplotlib import test|} +{|diagnostic:pyplot|}"; + + await TestCodeActionAsync( + markup, + title: "from matplotlib import pyplot as plt, test", + newText: "from matplotlib import pyplot as plt, test", + abbreviation: "plt", + relativePaths: new string[] { @"matplotlib\pyplot.py", @"matplotlib\test.py" }); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SuggestReverseAbbreviationForKnownModule() { + await TestCodeActionAsync( + @"{|insertionSpan:|}{|diagnostic:pd|}", + title: "import pandas as pd", + newText: "import pandas as pd" + Environment.NewLine + Environment.NewLine, + abbreviation: "pd", + relativePaths: "pandas.py"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SuggestReverseAbbreviationForKnownModule2() { + await TestCodeActionAsync( + @"{|insertionSpan:|}{|diagnostic:plt|}", + title: "from matplotlib import pyplot as plt", + newText: "from matplotlib import pyplot as plt" + Environment.NewLine + Environment.NewLine, + abbreviation: "plt", + relativePaths: @"matplotlib\pyplot.py"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task SuggestReverseAbbreviationForKnownModule3() { + var markup = @" +{|insertionSpan:from matplotlib import test|} +{|diagnostic:plt|}"; + + await TestCodeActionAsync( + markup, + title: "from matplotlib import pyplot as plt, test", + newText: "from matplotlib import pyplot as plt, test", + abbreviation: "plt", + relativePaths: new string[] { @"matplotlib\pyplot.py", @"matplotlib\test.py" }); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task AbbreviationConflict() { + var markup = @"{|insertionSpan:|}pd = 1 + +{|diagnostic:pandas|}"; + + await TestCodeActionAsync( + markup, + title: "import pandas as pd1", + newText: "import pandas as pd1" + Environment.NewLine + Environment.NewLine, + abbreviation: "pd1", + relativePaths: "pandas.py"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task AbbreviationConflict2() { + var markup = @"{|insertionSpan:|}{|diagnostic:pandas|} + +def Method(): + pd = 1"; + + await TestCodeActionAsync( + markup, + title: "import pandas as pd1", + newText: "import pandas as pd1" + Environment.NewLine + Environment.NewLine, + abbreviation: "pd1", + relativePaths: "pandas.py"); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task ContextBasedSuggestion() { + var markup = + @"from os import path +{|insertionSpan:|} +{|diagnostic:socket|}()"; + + var (analysis, codeActions, insertionSpan) = + await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + + codeActions.Should().NotContain(c => c.title == "import socket"); + + var title = "from socket import socket"; + var newText = "from socket import socket" + Environment.NewLine; + + var codeAction = codeActions.Single(c => c.title == title); + TestCodeAction(analysis.Document.Uri, codeAction, title, insertionSpan, newText); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task ValidToBeUsedInImport() { + await TestCodeActionAsync( + @"from os import path +{|insertionSpan:|} +{|diagnostic:join|}", + title: "from os.path import join", + newText: "from os.path import join" + Environment.NewLine); + } + + [TestMethod, Priority(0), Timeout(AnalysisTimeoutInMS)] + public async Task Disabled() { + var markup = @"from os import path +[|socket|]()"; + + MarkupUtils.GetSpan(markup, out var code, out var span); + + var analysis = await GetAnalysisAsync(code); + var diagnostics = GetDiagnostics(analysis, span.ToSourceSpan(analysis.Ast), MissingImportCodeActionProvider.Instance.FixableDiagnostics); + diagnostics.Should().NotBeEmpty(); + + var codeActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync(analysis, CodeActionSettings.Default, diagnostics, CancellationToken); + codeActions.Should().NotBeEmpty(); + + var emptyActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync( + analysis, new CodeActionSettings(null, new Dictionary() { { "addimports", false } }), diagnostics, CancellationToken); + emptyActions.Should().BeEmpty(); + } + + private async Task TestCodeActionAsync(string markup, string title, string newText, bool enableIndexManager = false) { + var (analysis, codeActions, insertionSpan) = + await GetAnalysisAndCodeActionsAndSpanAsync(markup, MissingImportCodeActionProvider.Instance.FixableDiagnostics, enableIndexManager); + + var codeAction = codeActions.Single(c => c.title == title); + TestCodeAction(analysis.Document.Uri, codeAction, title, insertionSpan, newText); + } + + private async Task<(IDocumentAnalysis analysis, CodeAction[] diagnostics, SourceSpan insertionSpan)> GetAnalysisAndCodeActionsAndSpanAsync( + string markup, IEnumerable codes, bool enableIndexManager = false) { + MarkupUtils.GetNamedSpans(markup, out var code, out var spans); + + var analysis = await GetAnalysisAsync(code); + + if (enableIndexManager) { + var serviceManager = (IServiceManager)analysis.ExpressionEvaluator.Services; + var indexManager = new IndexManager( + serviceManager.GetService(), + analysis.Document.Interpreter.LanguageVersion, + rootPath: null, + Array.Empty(), + Array.Empty(), + serviceManager.GetService()); + + // make sure index is done + await indexManager.IndexSnapshot(analysis.Document.Interpreter.ModuleResolution.CurrentPathResolver); + + serviceManager.AddService(indexManager); + } + + var insertionSpan = spans["insertionSpan"].First().ToSourceSpan(analysis.Ast); + + var diagnostics = GetDiagnostics(analysis, spans["diagnostic"].First().ToSourceSpan(analysis.Ast), codes); + var codeActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync(analysis, CodeActionSettings.Default, diagnostics, CancellationToken); + return (analysis, codeActions.ToArray(), insertionSpan); + } + + private static void TestCodeAction(Uri uri, CodeAction codeAction, string title, Core.Text.Range insertedSpan, string newText) { + codeAction.title.Should().Be(title); + codeAction.edit.changes.Should().HaveCount(1); + + var edit = codeAction.edit.changes[uri]; + edit.Single().range.Should().Be(insertedSpan); + edit.Single().newText.Should().Be(newText); + } + + private static Diagnostic[] GetDiagnostics(IDocumentAnalysis analysis, SourceSpan span, IEnumerable codes) { + var analyzer = analysis.ExpressionEvaluator.Services.GetService(); + return analyzer.LintModule(analysis.Document) + .Where(d => d.SourceSpan == span && codes.Any(e => string.Equals(e, d.ErrorCode))) + .Select(d => d.ToDiagnostic()) + .ToArray(); + } + + private async Task TestCodeActionAsync(string markup, string title, string newText, string abbreviation, params string[] relativePaths) { + MarkupUtils.GetNamedSpans(markup, out var code, out var spans); + + // get main analysis and add mock modules + var analysis = await GetAnalysisAsync(code); + + foreach (var relativePath in relativePaths) { + await GetAnalysisAsync("", analysis.ExpressionEvaluator.Services, modulePath: TestData.GetTestSpecificPath(relativePath)); + } + + // calculate actions + var diagnosticSpan = spans["diagnostic"].First().ToSourceSpan(analysis.Ast); + var diagnostics = GetDiagnostics(analysis, diagnosticSpan, MissingImportCodeActionProvider.Instance.FixableDiagnostics); + var codeActions = await new QuickFixCodeActionSource(analysis.ExpressionEvaluator.Services).GetCodeActionsAsync(analysis, CodeActionSettings.Default, diagnostics, CancellationToken); + + // verify results + var codeAction = codeActions.Single(c => c.title == title); + codeAction.edit.changes.Should().HaveCount(1); + + var edits = codeAction.edit.changes[analysis.Document.Uri]; + edits.Should().HaveCount(2); + + var invocationEdit = edits.Single(e => e.newText == abbreviation); + invocationEdit.range.Should().Be(diagnosticSpan); + + var insertEdit = edits.Single(e => e.newText == newText); + insertEdit.range.Should().Be(spans["insertionSpan"].First().ToSourceSpan(analysis.Ast)); + } + } +} diff --git a/src/LanguageServer/Test/ReferencesTests.cs b/src/LanguageServer/Test/ReferencesTests.cs index 405808b5c..7db878718 100644 --- a/src/LanguageServer/Test/ReferencesTests.cs +++ b/src/LanguageServer/Test/ReferencesTests.cs @@ -435,5 +435,54 @@ public async Task EmptyAnalysis() { var references = await rs.FindAllReferencesAsync(null, new SourceLocation(1, 1), ReferenceSearchOptions.All); references.Should().BeEmpty(); } + + [TestMethod, Priority(0)] + public async Task DelStatement1() { + const string code = @" +def func(a, b): + return a+b + +del func +"; + var analysis = await GetAnalysisAsync(code); + var rs = new ReferenceSource(Services); + var refs = await rs.FindAllReferencesAsync(analysis.Document.Uri, new SourceLocation(5, 6), ReferenceSearchOptions.All); + + refs.Should().HaveCount(2); + refs[0].range.Should().Be(1, 4, 1, 8); + refs[0].uri.Should().Be(analysis.Document.Uri); + refs[1].range.Should().Be(4, 4, 4, 8); + refs[1].uri.Should().Be(analysis.Document.Uri); + } + + [TestMethod, Priority(0)] + public async Task DelStatement2() { + const string code = @" +def func1(a, b): + return a+b + +def func2(a, b): + return a+b + +del (func1, func2) +"; + var analysis = await GetAnalysisAsync(code); + var rs = new ReferenceSource(Services); + var refs = await rs.FindAllReferencesAsync(analysis.Document.Uri, new SourceLocation(8, 6), ReferenceSearchOptions.All); + + refs.Should().HaveCount(2); + refs[0].range.Should().Be(1, 4, 1, 9); + refs[0].uri.Should().Be(analysis.Document.Uri); + refs[1].range.Should().Be(7, 5, 7, 10); + refs[1].uri.Should().Be(analysis.Document.Uri); + + refs = await rs.FindAllReferencesAsync(analysis.Document.Uri, new SourceLocation(8, 14), ReferenceSearchOptions.All); + + refs.Should().HaveCount(2); + refs[0].range.Should().Be(4, 4, 4, 9); + refs[0].uri.Should().Be(analysis.Document.Uri); + refs[1].range.Should().Be(7, 12, 7, 17); + refs[1].uri.Should().Be(analysis.Document.Uri); + } } } diff --git a/src/LanguageServer/Test/RenameTests.cs b/src/LanguageServer/Test/RenameTests.cs index 6b6bcb2f5..e16649883 100644 --- a/src/LanguageServer/Test/RenameTests.cs +++ b/src/LanguageServer/Test/RenameTests.cs @@ -220,7 +220,7 @@ public async Task NoRenameInCompiled() { } [TestMethod, Priority(0)] - public async Task NoRenameInLibrary() { + public async Task NoRenameInSystemLibrary() { const string code = @"from logging import BASIC_FORMAT"; var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); var rs = new RenameSource(Services); @@ -231,5 +231,18 @@ public async Task NoRenameInLibrary() { wse = await rs.RenameAsync(analysis.Document.Uri, new SourceLocation(1, 23), "z"); wse.Should().BeNull(); } + + [TestMethod, Priority(0)] + public async Task NoRenameInUserLibrary() { + const string code = @" +import Library1 +Library1.make_foo() +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); + var rs = new RenameSource(Services); + + var wse = await rs.RenameAsync(analysis.Document.Uri, new SourceLocation(3, 16), "z"); + wse.Should().BeNull(); + } } } diff --git a/src/LanguageServer/Test/SignatureTests.cs b/src/LanguageServer/Test/SignatureTests.cs index eb098cea1..84aea66dc 100644 --- a/src/LanguageServer/Test/SignatureTests.cs +++ b/src/LanguageServer/Test/SignatureTests.cs @@ -336,5 +336,39 @@ def get(self) -> List['X']: sig.signatures.Length.Should().Be(1); sig.signatures[0].label.Should().Be("get() -> List[X]"); } + + [TestMethod, Priority(0)] + public async Task PositionalOnly() { + const string code = @" +def f(a, b, /, c, d, *, e, f): + pass + +f() +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); + var src = new SignatureSource(new PlainTextDocumentationSource()); + + var sig = src.GetSignature(analysis, new SourceLocation(5, 3)); + sig.signatures.Should().NotBeNull(); + sig.signatures.Length.Should().Be(1); + sig.signatures[0].label.Should().Be("f(a, b, /, c, d, *, e, f)"); + } + + [TestMethod, Priority(0)] + public async Task DefaultValues() { + const string code = @" +def hello(a, b=100, c={'a':4}): + return a + b + +hello() +"; + var analysis = await GetAnalysisAsync(code, PythonVersions.LatestAvailable3X); + var src = new SignatureSource(new PlainTextDocumentationSource()); + + var sig = src.GetSignature(analysis, new SourceLocation(5, 7)); + sig.signatures.Should().NotBeNull(); + sig.signatures.Length.Should().Be(1); + sig.signatures[0].label.Should().Be("hello(a, b=100, c={'a':4})"); + } } } diff --git a/src/LanguageServer/Test/SymbolIndexTests.cs b/src/LanguageServer/Test/SymbolIndexTests.cs index 861eb1cec..b895a52f3 100644 --- a/src/LanguageServer/Test/SymbolIndexTests.cs +++ b/src/LanguageServer/Test/SymbolIndexTests.cs @@ -160,6 +160,47 @@ public async Task IndexWorkspaceSymbolsCaseInsensitiveAsync() { } } + [TestMethod, Priority(0)] + public async Task IndexWorkspaceSymbolsFuzzyAsync() { + const string code = @"class FXoo(object): + def foxo(self, x): ..."; + + using (var index = MakeSymbolIndex()) { + var path = TestData.GetDefaultModulePath(); + index.Add(path, DocumentWithAst(code)); + + var symbols = await index.WorkspaceSymbolsAsync("foo", maxSymbols); + symbols.Should().BeEquivalentToWithStrictOrdering(new[] { + new FlatSymbol("FXoo", SymbolKind.Class, path, new SourceSpan(1, 7, 1, 11)), + new FlatSymbol("foxo", SymbolKind.Method, path, new SourceSpan(2, 9, 2, 13), "FXoo"), + }); + } + } + + [TestMethod, Priority(0)] + public async Task IndexWorkspaceSymbolsOrderingAsync() { + const string code = @"class FXoo(object): + def foxo(self, x): ... + +class Foo: ... + +def foobar(): ... +"; + + using (var index = MakeSymbolIndex()) { + var path = TestData.GetDefaultModulePath(); + index.Add(path, DocumentWithAst(code)); + + var symbols = await index.WorkspaceSymbolsAsync("foo", maxSymbols); + symbols.Should().BeEquivalentToWithStrictOrdering(new[] { + new FlatSymbol("Foo", SymbolKind.Class, path, new SourceSpan(4, 7, 4, 10)), + new FlatSymbol("foobar", SymbolKind.Function, path, new SourceSpan(6, 5, 6, 11)), + new FlatSymbol("FXoo", SymbolKind.Class, path, new SourceSpan(1, 7, 1, 11)), + new FlatSymbol("foxo", SymbolKind.Method, path, new SourceSpan(2, 9, 2, 13), "FXoo"), + }); + } + } + [TestMethod, Priority(0)] public void MarkAsPendingWaitsForUpdates() { using (var index = MakeSymbolIndex()) { diff --git a/src/LanguageServer/Test/SymbolIndexWalkerTests.cs b/src/LanguageServer/Test/SymbolIndexWalkerTests.cs index f4fb90fee..89a74ce87 100644 --- a/src/LanguageServer/Test/SymbolIndexWalkerTests.cs +++ b/src/LanguageServer/Test/SymbolIndexWalkerTests.cs @@ -196,14 +196,7 @@ from os.path import ( join as osjoin2, exists as osexists, expanduser ) "; var symbols = WalkSymbols(code); - symbols.Should().BeEquivalentToWithStrictOrdering(new[] { - new HierarchicalSymbol("sys", SymbolKind.Module, new SourceSpan(1, 8, 1, 11)), - new HierarchicalSymbol("np", SymbolKind.Module, new SourceSpan(2, 17, 2, 19)), - new HierarchicalSymbol("osjoin", SymbolKind.Module, new SourceSpan(3, 29, 3, 35)), - new HierarchicalSymbol("osjoin2", SymbolKind.Module, new SourceSpan(4, 31, 4, 38)), - new HierarchicalSymbol("osexists", SymbolKind.Module, new SourceSpan(4, 50, 4, 58)), - new HierarchicalSymbol("expanduser", SymbolKind.Module, new SourceSpan(4, 60, 4, 70)), - }); + symbols.Should().BeEmpty(); } [TestMethod, Priority(0)] @@ -657,6 +650,14 @@ public void WalkerNamedExpression() { }); } + [TestMethod, Priority(0)] + public void WalkerNoNameFunction() { + var code = @"def ():"; + + var symbols = WalkSymbols(code); + symbols.Should().BeEmpty(); + } + private PythonAst GetParse(string code, PythonLanguageVersion version) => Parser.CreateParser(new StringReader(code), version).ParseFile(); diff --git a/src/LanguageServer/Test/UniqueNameGeneratorTests.cs b/src/LanguageServer/Test/UniqueNameGeneratorTests.cs new file mode 100644 index 000000000..4233ae81f --- /dev/null +++ b/src/LanguageServer/Test/UniqueNameGeneratorTests.cs @@ -0,0 +1,151 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Reflection.PortableExecutable; +using System.Threading.Tasks; +using FluentAssertions; +using Microsoft.Python.Analysis; +using Microsoft.Python.LanguageServer.Tests.FluentAssertions; +using Microsoft.Python.LanguageServer.Utilities; +using Microsoft.Python.Parsing.Tests; +using Microsoft.Python.UnitTests.Core; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using TestUtilities; + +namespace Microsoft.Python.LanguageServer.Tests { + [TestClass] + public class UniqueNameGeneratorTests : LanguageServerTestBase { + public TestContext TestContext { get; set; } + + [TestInitialize] + public void TestInitialize() + => TestEnvironmentImpl.TestInitialize($"{TestContext.FullyQualifiedTestClassName}.{TestContext.TestName}"); + + [TestCleanup] + public void Cleanup() => TestEnvironmentImpl.TestCleanup(); + + [TestMethod, Priority(0)] + public async Task NoConflict() { + MarkupUtils.GetPosition(@"$$", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name"); + Test(analysis, "name", "name"); + } + + [TestMethod, Priority(0)] + public async Task Conflict_TopLevel() { + MarkupUtils.GetPosition(@"$$ + +name = 1", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name1"); + Test(analysis, "name", "name1"); + } + + [TestMethod, Priority(0)] + public async Task Conflict_TopLevel2() { + MarkupUtils.GetPosition(@"$$ + +class name: + pass", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name1"); + Test(analysis, "name", "name1"); + } + + [TestMethod, Priority(0)] + public async Task Conflict_Function() { + MarkupUtils.GetPosition(@"def Test(): + $$ + +name = 1", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name1"); + Test(analysis, "name", "name1"); + } + + [TestMethod, Priority(0)] + public async Task Conflict_Function2() { + MarkupUtils.GetPosition(@"def Test(): + name = 1 + $$ + pass", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name1"); + Test(analysis, "name", "name1"); + } + + [TestMethod, Priority(0)] + public async Task Conflict_Function3() { + MarkupUtils.GetPosition(@"def Test(): + name = 1 + +def Test2(): + $$ + pass", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name"); + Test(analysis, "name", "name1"); + } + + [TestMethod, Priority(0)] + public async Task Conflict_TopLevel3() { + MarkupUtils.GetPosition(@"def Test(): + name = 1 + +$$", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name"); + Test(analysis, "name", "name1"); + } + + [TestMethod, Priority(0)] + public async Task MultipleConflicts() { + MarkupUtils.GetPosition(@" +name1 = 1 + +class name3: + name2 = 1 + +def Test(): + name = 1 + + def name4(): + pass + +$$", out var code, out int position); + + var analysis = await GetAnalysisAsync(code); + Test(analysis, position, "name", "name"); + Test(analysis, "name", "name5"); + } + + private static void Test(IDocumentAnalysis analysis, int position, string name, string expected) { + var actual = UniqueNameGenerator.Generate(analysis, position, name); + actual.Should().Be(expected); + } + + private static void Test(IDocumentAnalysis analysis, string name, string expected) { + Test(analysis, position: -1, name, expected); + } + } +} diff --git a/src/PLS.sln b/src/PLS.sln index 2bceb6b86..ab969a558 100644 --- a/src/PLS.sln +++ b/src/PLS.sln @@ -27,10 +27,6 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Python.Analysis.T EndProject Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Python.LanguageServer.Tests", "LanguageServer\Test\Microsoft.Python.LanguageServer.Tests.csproj", "{3BAB87E1-79FD-45D1-8564-CAF87D4D16CA}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Python.Analysis.Caching", "Caching\Impl\Microsoft.Python.Analysis.Caching.csproj", "{42BD3C80-3E57-4847-8142-84F6B682EA8D}" -EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Python.Analysis.Caching.Tests", "Caching\Test\Microsoft.Python.Analysis.Caching.Tests.csproj", "{40CD3A74-B0B6-4A37-AE65-5B203C38D0E2}" -EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -77,14 +73,6 @@ Global {3BAB87E1-79FD-45D1-8564-CAF87D4D16CA}.Debug|Any CPU.Build.0 = Debug|Any CPU {3BAB87E1-79FD-45D1-8564-CAF87D4D16CA}.Release|Any CPU.ActiveCfg = Release|Any CPU {3BAB87E1-79FD-45D1-8564-CAF87D4D16CA}.Release|Any CPU.Build.0 = Release|Any CPU - {42BD3C80-3E57-4847-8142-84F6B682EA8D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {42BD3C80-3E57-4847-8142-84F6B682EA8D}.Debug|Any CPU.Build.0 = Debug|Any CPU - {42BD3C80-3E57-4847-8142-84F6B682EA8D}.Release|Any CPU.ActiveCfg = Release|Any CPU - {42BD3C80-3E57-4847-8142-84F6B682EA8D}.Release|Any CPU.Build.0 = Release|Any CPU - {40CD3A74-B0B6-4A37-AE65-5B203C38D0E2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {40CD3A74-B0B6-4A37-AE65-5B203C38D0E2}.Debug|Any CPU.Build.0 = Debug|Any CPU - {40CD3A74-B0B6-4A37-AE65-5B203C38D0E2}.Release|Any CPU.ActiveCfg = Release|Any CPU - {40CD3A74-B0B6-4A37-AE65-5B203C38D0E2}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -100,8 +88,6 @@ Global {2C8DE250-41F4-4FC5-A661-76E2A4172891} = {C465393D-145E-4695-A7DB-AF55951BD533} {D8D85896-5DB0-4FA6-B744-910A272C39F9} = {80AA38A1-3E82-4B87-BB21-FDEDD2CC87E6} {3BAB87E1-79FD-45D1-8564-CAF87D4D16CA} = {80AA38A1-3E82-4B87-BB21-FDEDD2CC87E6} - {42BD3C80-3E57-4847-8142-84F6B682EA8D} = {C465393D-145E-4695-A7DB-AF55951BD533} - {40CD3A74-B0B6-4A37-AE65-5B203C38D0E2} = {80AA38A1-3E82-4B87-BB21-FDEDD2CC87E6} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {ABC12ED7-0EC8-4219-8A14-A058F7942D92} diff --git a/src/Parsing/Impl/Ast/AssignmentStatement.cs b/src/Parsing/Impl/Ast/AssignmentStatement.cs index 09d22059e..4724470be 100644 --- a/src/Parsing/Impl/Ast/AssignmentStatement.cs +++ b/src/Parsing/Impl/Ast/AssignmentStatement.cs @@ -38,7 +38,9 @@ public override IEnumerable GetChildNodes() { foreach (var expression in _left) { yield return expression; } - if (Right != null) yield return Right; + if (Right != null) { + yield return Right; + } } public override void Walk(PythonWalker walker) { diff --git a/src/Parsing/Impl/Ast/CallExpression.cs b/src/Parsing/Impl/Ast/CallExpression.cs index 9a5231862..00522512f 100644 --- a/src/Parsing/Impl/Ast/CallExpression.cs +++ b/src/Parsing/Impl/Ast/CallExpression.cs @@ -17,7 +17,6 @@ using System.Text; using System.Threading; using System.Threading.Tasks; -using Microsoft.Python.Core; using Microsoft.Python.Core.Collections; namespace Microsoft.Python.Parsing.Ast { diff --git a/src/Parsing/Impl/Ast/NamedExpressionErrorWalker.cs b/src/Parsing/Impl/Ast/NamedExpressionErrorWalker.cs new file mode 100644 index 000000000..dae5f8a68 --- /dev/null +++ b/src/Parsing/Impl/Ast/NamedExpressionErrorWalker.cs @@ -0,0 +1,216 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Core; +using Microsoft.Python.Core.Collections; +using Microsoft.Python.Core.Disposables; +using Microsoft.Python.Parsing.Extensions; + +namespace Microsoft.Python.Parsing.Ast { + internal class NamedExpressionErrorWalker : PythonWalker { + private readonly Action _reportError; + + private NameScope _scope = NameScope.Root; + private bool _insideForList = false; + + internal static void Check(PythonAst ast, PythonLanguageVersion langVersion, Action reportError) { + if (langVersion < PythonLanguageVersion.V38) { + return; + } + + ast.Walk(new NamedExpressionErrorWalker(reportError)); + } + + private NamedExpressionErrorWalker(Action reportError) { + _reportError = reportError; + } + + public override bool Walk(ClassDefinition node) { + _scope = new NameScope(_scope, true); + return base.Walk(node); + } + + public override void PostWalk(ClassDefinition node) { + base.PostWalk(node); + _scope = _scope.Prev; + } + + public override bool Walk(FunctionDefinition node) { + // LambdaExpression wraps FunctionDefinition, so this handles both functions and lambdas. + var parameters = node.Parameters.Select(p => p.Name).Where(IsNotIgnoredName); + _scope = new NameScope(_scope, parameters); + return base.Walk(node); + } + + public override void PostWalk(FunctionDefinition node) { + base.PostWalk(node); + _scope = _scope.Prev; + } + + public override bool Walk(GeneratorExpression node) { + using (ComprehensionScope()) { + foreach (var ci in node.Iterators) { + ci.Walk(this); + } + node.Item?.Walk(this); + } + return false; + } + + public override bool Walk(DictionaryComprehension node) { + using (ComprehensionScope()) { + foreach (var ci in node.Iterators) { + ci.Walk(this); + } + node.Key?.Walk(this); + node.Value?.Walk(this); + } + return false; + } + + public override bool Walk(ListComprehension node) { + using (ComprehensionScope()) { + foreach (var ci in node.Iterators) { + ci.Walk(this); + } + node.Item?.Walk(this); + } + return false; + } + + public override bool Walk(SetComprehension node) { + using (ComprehensionScope()) { + foreach (var ci in node.Iterators) { + ci.Walk(this); + } + node.Item?.Walk(this); + } + return false; + } + + public override bool Walk(ComprehensionFor node) { + var names = node.Left?.ChildNodesBreadthFirst().OfType().Where(IsNotIgnoredName).MaybeEnumerate(); + _scope.AddIterators(names.Select(name => name.Name).ToImmutableArray()); + + // Collect this ahead of time, so that walking the list does not modify the list of names. + var bad = names.Where(name => _scope.IsNamed(name.Name)).ToImmutableArray(); + + var old = _insideForList; + _insideForList = true; + node.List?.Walk(this); + _insideForList = old; + + foreach (var name in bad) { + ReportSyntaxError(name, Resources.NamedExpressionIteratorRebindsNamedErrorMsg.FormatInvariant(name.Name)); + } + + return false; + } + + public override bool Walk(NamedExpression node) { + var names = node.Target.ChildNodesBreadthFirst().OfType().Where(IsNotIgnoredName).MaybeEnumerate(); + _scope.AddNamed(names.Select(name => name.Name).ToImmutableArray()); + + if (_insideForList) { + ReportSyntaxError(node, Resources.NamedExpressionInComprehensionIteratorErrorMsg); + return false; + } + + if (_scope.IsClassTarget) { + ReportSyntaxError(node, Resources.NamedExpressionInClassBodyErrorMsg); + return false; + } + + foreach (var name in names) { + if (_scope.IsIterator(name.Name)) { + ReportSyntaxError(name, Resources.NamedExpressionRebindIteratorErrorMsg.FormatInvariant(name.Name)); + } + } + + node.Value?.Walk(this); + return false; + } + + private void ReportSyntaxError(Node node, string message) => _reportError(node.StartIndex, node.EndIndex, message); + + private static bool IsNotIgnoredName(NameExpression name) => IsNotIgnoredName(name.Name); + + private static bool IsNotIgnoredName(string name) => !string.IsNullOrWhiteSpace(name) && name != "_"; + + private IDisposable ComprehensionScope() { + _scope = new NameScope(_scope); + return Disposable.Create(() => _scope = _scope.Prev); + } + + private class NameScope { + public static NameScope Root = new NameScope(); + + private readonly bool _isRoot; + private readonly bool? _isClassTarget; + private readonly ImmutableArray _funcParams = ImmutableArray.Empty; + + private ImmutableArray _named = ImmutableArray.Empty; + private ImmutableArray _iterators = ImmutableArray.Empty; + + private NameScope() { + _isRoot = true; + } + + public NameScope(NameScope prev, bool? isClassTarget = null) { + Prev = prev; + _isClassTarget = isClassTarget; + } + + public NameScope(NameScope prev, IEnumerable funcParams) : this(prev, false) { + _funcParams = funcParams.ToImmutableArray(); + } + + public NameScope Prev { get; } = null; + + public bool IsClassTarget => _isClassTarget ?? Prev?.IsClassTarget ?? false; + + public void AddIterators(ImmutableArray names) { + if (!_isRoot) { + _iterators = _iterators.AddRange(names); + } + } + + public void AddNamed(ImmutableArray names) { + if (!_isRoot) { + _named = _named.AddRange(names); + } + } + + public bool IsIterator(string name) { + if (_isRoot || _funcParams.Contains(name)) { + return false; + } + + return _iterators.Contains(name) || (Prev?.IsIterator(name) ?? false); + } + + public bool IsNamed(string name) { + if (_isRoot || _funcParams.Contains(name)) { + return false; + } + + return _named.Contains(name) || (Prev?.IsNamed(name) ?? false); + } + } + } +} diff --git a/src/Parsing/Impl/Ast/Parameter.cs b/src/Parsing/Impl/Ast/Parameter.cs index 080c5aa2c..90234c4da 100644 --- a/src/Parsing/Impl/Ast/Parameter.cs +++ b/src/Parsing/Impl/Ast/Parameter.cs @@ -52,7 +52,11 @@ public Parameter(NameExpression name, ParameterKind kind) { public bool IsKeywordOnly => Kind == ParameterKind.KeywordOnly; - public ParameterKind Kind { get; } + public bool IsPositionalOnlyMarker => Kind == ParameterKind.PositionalOnlyMarker; + + public bool IsPositionalOnly => Kind == ParameterKind.PositionalOnly; + + public ParameterKind Kind { get; internal set; } public override IEnumerable GetChildNodes() { if (NameExpression != null) yield return NameExpression; @@ -116,6 +120,7 @@ internal override void AppendCodeString(StringBuilder res, PythonAst ast, CodeFo leadingWhiteSpace = null; res.Append('*'); break; + case ParameterKind.PositionalOnly: case ParameterKind.Normal: if (this.IsAltForm(ast)) { res.Append(leadingWhiteSpace ?? this.GetPreceedingWhiteSpaceDefaultNull(ast) ?? string.Empty); @@ -131,6 +136,11 @@ internal override void AppendCodeString(StringBuilder res, PythonAst ast, CodeFo break; case ParameterKind.KeywordOnly: break; + case ParameterKind.PositionalOnlyMarker: + res.Append(leadingWhiteSpace ?? this.GetPreceedingWhiteSpaceDefaultNull(ast) ?? string.Empty); + leadingWhiteSpace = null; + res.Append('/'); + break; default: throw new InvalidOperationException(); } diff --git a/src/Parsing/Impl/Ast/ParameterKind.cs b/src/Parsing/Impl/Ast/ParameterKind.cs index 57d2c3298..c56217423 100644 --- a/src/Parsing/Impl/Ast/ParameterKind.cs +++ b/src/Parsing/Impl/Ast/ParameterKind.cs @@ -15,9 +15,11 @@ namespace Microsoft.Python.Parsing.Ast { public enum ParameterKind { - Normal, - List, - Dictionary, - KeywordOnly + Normal, // Positional or keyword + List, // *args, or just '*' with a null name for a marker + Dictionary, // **kwargs + KeywordOnly, // after '*' + PositionalOnly, // Before '/' + PositionalOnlyMarker // '/' marker itself } } diff --git a/src/Parsing/Impl/Ast/SourceLocationExtensions.cs b/src/Parsing/Impl/Ast/SourceLocationExtensions.cs index 9377fe70f..157c9b127 100644 --- a/src/Parsing/Impl/Ast/SourceLocationExtensions.cs +++ b/src/Parsing/Impl/Ast/SourceLocationExtensions.cs @@ -19,17 +19,34 @@ namespace Microsoft.Python.Parsing.Ast { public static class SourceLocationExtensions { public static int ToIndex(this SourceLocation location, ILocationConverter lc) => lc.LocationToIndex(location); + + public static SourceLocation ToSourceLocation(this Position position, ILocationConverter lc = null) { + var location = new SourceLocation(position.line + 1, position.character + 1); + if (lc == null) { + return location; + } + + return new SourceLocation(lc.LocationToIndex(location), location.Line, location.Column); + } + } + + public static class RangeExtensions { + public static IndexSpan ToIndexSpan(this Range range, ILocationConverter lc) + => IndexSpan.FromBounds(lc.LocationToIndex(range.start), lc.LocationToIndex(range.end)); + public static SourceSpan ToSourceSpan(this Range range, ILocationConverter lc = null) + => new SourceSpan(range.start.ToSourceLocation(lc), range.end.ToSourceLocation(lc)); } public static class SourceSpanExtensions { public static IndexSpan ToIndexSpan(this SourceSpan span, ILocationConverter lc) => IndexSpan.FromBounds(lc.LocationToIndex(span.Start), lc.LocationToIndex(span.End)); - public static IndexSpan ToIndexSpan(this Range range, ILocationConverter lc) - => IndexSpan.FromBounds(lc.LocationToIndex(range.start), lc.LocationToIndex(range.end)); } public static class IndexSpanExtensions { public static SourceSpan ToSourceSpan(this IndexSpan span, ILocationConverter lc) => lc != null ? new SourceSpan(lc.IndexToLocation(span.Start), lc.IndexToLocation(span.End)) : default; + public static bool Contains(this IndexSpan span, IndexSpan other) { + return span.Start <= other.Start && other.End <= span.End; + } } } diff --git a/src/Parsing/Impl/Ast/TypeAnnotation.cs b/src/Parsing/Impl/Ast/TypeAnnotation.cs index b7d387cde..b8f10ca4c 100644 --- a/src/Parsing/Impl/Ast/TypeAnnotation.cs +++ b/src/Parsing/Impl/Ast/TypeAnnotation.cs @@ -27,9 +27,6 @@ public TypeAnnotation(PythonLanguageVersion version, Expression expr) { Expression = expr ?? throw new ArgumentNullException(nameof(expr)); } - public static TypeAnnotation FromType(TypeAnnotationConverter converter, T type) where T : class - => throw new NotImplementedException(); - public PythonLanguageVersion LanguageVersion { get; } public Expression Expression { get; } @@ -81,6 +78,7 @@ public T GetResult(TypeAnnotationConverter converter) where T : class { } public override bool Walk(ConstantExpression node) { + // TODO: Do not perform recursive string parsing. switch (node.Value) { case string s: _parse(s)?.Walk(this); @@ -91,6 +89,9 @@ public override bool Walk(ConstantExpression node) { case null: _ops.Add(new NameOp { Name = "None" }); break; + default: + _ops.Add(new MakeUnknownOp()); + break; } return false; @@ -277,6 +278,13 @@ public override bool Apply(TypeAnnotationConverter converter, Stack(TypeAnnotationConverter converter, Stack> stack) { + stack.Push(new KeyValuePair(null, default)); + return true; + } + } } } diff --git a/src/Parsing/Impl/Extensions/AstExtensions.cs b/src/Parsing/Impl/Extensions/AstExtensions.cs new file mode 100644 index 000000000..94933178b --- /dev/null +++ b/src/Parsing/Impl/Extensions/AstExtensions.cs @@ -0,0 +1,35 @@ +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System.Collections.Generic; +using System.Linq; +using Microsoft.Python.Core; +using Microsoft.Python.Parsing.Ast; + +namespace Microsoft.Python.Parsing.Extensions { + public static class AstExtensions { + public static IEnumerable ChildNodesDepthFirst(this Node node) + => node.TraverseDepthFirst(n => n.GetChildNodes()); + + public static IEnumerable SelectChildNodesDepthFirst(this IEnumerable nodes) + => nodes.SelectMany(n => n.ChildNodesDepthFirst()); + + public static IEnumerable ChildNodesBreadthFirst(this Node node) + => node.TraverseBreadthFirst(n => n.GetChildNodes()); + + public static IEnumerable SelectChildNodesBreadthFirst(this IEnumerable nodes) + => nodes.SelectMany(n => n.ChildNodesBreadthFirst()); + } +} diff --git a/src/Parsing/Impl/FStringParser.cs b/src/Parsing/Impl/FStringParser.cs index 6609ec777..cbad7ca21 100644 --- a/src/Parsing/Impl/FStringParser.cs +++ b/src/Parsing/Impl/FStringParser.cs @@ -32,7 +32,6 @@ internal class FStringParser { private static readonly StringSpan DoubleOpen = new StringSpan("{{"); private static readonly StringSpan DoubleClose = new StringSpan("}}"); private static readonly StringSpan NotEqual = new StringSpan("!="); - private static readonly StringSpan EqualEqual = new StringSpan("=="); private static readonly StringSpan BackslashN = new StringSpan("\\N"); internal FStringParser(List fStringChildren, string fString, bool isRaw, @@ -136,7 +135,7 @@ private Node ParseFStringExpression() { _buffer.Clear(); } - Debug.Assert(CurrentChar == '}' || CurrentChar == '!' || CurrentChar == ':' || (_langVersion >= PythonLanguageVersion.V38 && CurrentChar == '=')); + Debug.Assert(CurrentChar == '}' || CurrentChar == '!' || CurrentChar == ':' || CurrentChar == '='); MaybeReadEqualSpecifier(); var conversion = MaybeReadConversionChar(); @@ -232,10 +231,23 @@ private void BufferInnerExpression() { while (!EndOfFString) { var ch = CurrentChar; + var appendExtra = false; + if (!quoteChar.HasValue && _nestedParens.Count == 0) { switch (ch) { - case '!' when !IsNext(NotEqual): - case '=' when !IsNext(EqualEqual) && _langVersion >= PythonLanguageVersion.V38: + case '=': + case '!': + if (!IsEqualsAfterNext) { + return; + } + appendExtra = true; + break; + + case '<': + case '>': + appendExtra = IsEqualsAfterNext; + break; + case '}': case ':': return; @@ -253,6 +265,10 @@ private void BufferInnerExpression() { } else { HandleInnerExprOutsideString(ref quoteChar, ref stringType); } + + if (appendExtra) { + _buffer.Append(NextChar()); + } } } @@ -484,5 +500,7 @@ private bool IsAsciiWhiteSpace { return false; } } + + private bool IsEqualsAfterNext => _position + 1 < _fString.Length && _fString[_position + 1] == '='; } } diff --git a/src/Parsing/Impl/Microsoft-Python-Parsing.nuspec b/src/Parsing/Impl/Microsoft-Python-Parsing.nuspec new file mode 100644 index 000000000..28e361cbc --- /dev/null +++ b/src/Parsing/Impl/Microsoft-Python-Parsing.nuspec @@ -0,0 +1,27 @@ + + + + Microsoft.Python.Parsing + $version$ + Microsoft Python Static Analysis + Microsoft + Microsoft + + https://github.com/Microsoft/python-language-server/blob/master/LICENSE + https://github.com/Microsoft/python-language-server + false + Microsoft Python Parsing Library + Initial release. + © Microsoft Corporation. All rights reserved. + Python + + + + + + + + + + + \ No newline at end of file diff --git a/src/Parsing/Impl/Microsoft.Python.Parsing.csproj b/src/Parsing/Impl/Microsoft.Python.Parsing.csproj index 0b21cd07b..e96194f50 100644 --- a/src/Parsing/Impl/Microsoft.Python.Parsing.csproj +++ b/src/Parsing/Impl/Microsoft.Python.Parsing.csproj @@ -9,7 +9,7 @@ 1701, 1702 - "You may need to supply assembly policy" --> 1701;1702;$(NoWarn) - 7.2 + true @@ -35,6 +35,11 @@ Resources.Designer.cs + + + PreserveNewest + + diff --git a/src/Parsing/Impl/Parser.cs b/src/Parsing/Impl/Parser.cs index 400b205d9..e392be1cd 100644 --- a/src/Parsing/Impl/Parser.cs +++ b/src/Parsing/Impl/Parser.cs @@ -246,7 +246,7 @@ private PythonAst CreateAst(Uri module, Statement ret) { ast.SetAttributes(_attributes); PythonNameBinder.BindAst(_langVersion, ast, _errors, _bindReferences); - + NamedExpressionErrorWalker.Check(ast, _langVersion, ReportSyntaxError); return ast; } @@ -595,7 +595,7 @@ private Statement ParseSmallStmt() { case TokenKind.KeywordContinue: if (!_inLoop) { ReportSyntaxError(Resources.ContinueNotInLoopErrorMsg);//'continue' not properly in loop - } else if (_inFinally) { + } else if (_inFinally && _langVersion < PythonLanguageVersion.V38) { ReportSyntaxError(Resources.ContinueNotSupportedInsideFinallyErrorMsg);//'continue' not supported inside 'finally' clause } return FinishSmallStmt(new ContinueStatement()); @@ -1982,12 +1982,14 @@ private FunctionDefinition ParseFuncDef(bool isCoroutine) { //varargslist: (fpdef ['=' expression ] ',')* ('*' NAME [',' '**' NAME] | '**' NAME) | fpdef ['=' expression] (',' fpdef ['=' expression])* [','] //fpdef: NAME | '(' fplist ')' //fplist: fpdef (',' fpdef)* [','] + // Not above: the 3.8+'s positional marker. private Parameter[] ParseVarArgsList(TokenKind terminator, bool allowAnnotations, out List commaWhiteSpace, out bool ateTerminator) { var parameters = new List(); commaWhiteSpace = MakeWhiteSpaceList(); var namedOnly = false; var lastComma = true; + int? posOnlyEnd = null; var lastStart = -1; @@ -2018,6 +2020,11 @@ private Parameter[] ParseVarArgsList(TokenKind terminator, bool allowAnnotations start = GetStart(); kind = ParameterKind.Dictionary; preStarWhitespace = _tokenWhiteSpace; + } else if (_langVersion >= PythonLanguageVersion.V38 && MaybeEat(TokenKind.Divide)) { + start = GetStart(); + posOnlyEnd = posOnlyEnd ?? pos; + kind = ParameterKind.PositionalOnlyMarker; + preStarWhitespace = _tokenWhiteSpace; } var name = TokenToName(PeekToken()); @@ -2030,8 +2037,7 @@ private Parameter[] ParseVarArgsList(TokenKind terminator, bool allowAnnotations AddPreceedingWhiteSpace(ne); } p = new Parameter(ne, kind); - } else if (kind == ParameterKind.List) { - // bare lists are allowed + } else if (kind == ParameterKind.List || kind == ParameterKind.PositionalOnlyMarker) { p = new Parameter(null, kind); } else { var expr = ParseExpression(); @@ -2065,10 +2071,29 @@ private Parameter[] ParseVarArgsList(TokenKind terminator, bool allowAnnotations } } + if (posOnlyEnd.HasValue) { + // Re-kind parameters before '/' as positional only. + for (var pos = 0; pos < posOnlyEnd.Value; pos++) { + var p = parameters[pos]; + // Not having the "Normal" kind means the '/' is misplaced, which will be errored below. + if (p.Kind == ParameterKind.Normal) { + p.Kind = ParameterKind.PositionalOnly; + } + } + } + // Now we validate the parameters - bool seenListArg = false, seenDictArg = false, seenDefault = false; + bool seenListArg = false, seenDictArg = false, seenDefault = false, seenPositional = false, first = true; var seenNames = new HashSet(); foreach (var p in parameters) { + var isFirst = first; + first = false; + + if (isFirst && p.IsPositionalOnlyMarker) { + ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalOnlyMarkerFirstParamErrorMessage); + continue; + } + if (p.Annotation != null) { if (!_stubFile && _langVersion.Is2x()) { ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.ParameterAnnotationsRequire3dotXErrorMsg);//invalid syntax, parameter annotations require 3.x @@ -2102,7 +2127,7 @@ private Parameter[] ParseVarArgsList(TokenKind terminator, bool allowAnnotations } if (string.IsNullOrEmpty(p.Name)) { - if (p.Kind != ParameterKind.List || !(_stubFile || _langVersion.Is3x())) { + if ((p.Kind != ParameterKind.List && p.Kind != ParameterKind.PositionalOnlyMarker) || !(_stubFile || _langVersion.Is3x())) { ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.InvalidSyntaxErrorMsg);//Invalid Syntax continue; } @@ -2116,10 +2141,23 @@ private Parameter[] ParseVarArgsList(TokenKind terminator, bool allowAnnotations } seenListArg = true; } else if (p.Kind == ParameterKind.Dictionary) { - if (seenDictArg) { + if (seenDictArg) { ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.DuplicateArgsDoubleArgumentErrorMsg);//duplicate ** args arguments } seenDictArg = true; + } else if (p.Kind == ParameterKind.PositionalOnlyMarker) { + if (seenPositional) { + ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalOnlyMarkerDuplicateErrorMsg); + } else if (seenListArg) { + ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalOnlyMarkerAfterListArgsErrorMsg); + } else if (seenDictArg) { + ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalOnlyMarkerAfterDictArgsErrorMsg); + } else if (p.Annotation != null) { + ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalOnlyMarkerAnnotationErrorMsg); + } else if (p.DefaultValue != null) { + ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalOnlyMarkerDefaultErrorMsg); + } + seenPositional = true; } else if (seenListArg && p.Kind != ParameterKind.KeywordOnly) { ReportSyntaxError(p.StartIndex, p.EndIndex, Resources.PositionalParameterNotAllowedErrorMsg);//positional parameter after * args not allowed } else if (seenDictArg) { @@ -4951,12 +4989,11 @@ public override void Add(string message, SourceSpan span, int errorCode, Severit } private Expression RemoveParenthesis(Expression expr) { - while(expr is ParenthesisExpression parenExpr) { + while (expr is ParenthesisExpression parenExpr) { expr = parenExpr.Expression; } return expr; } - #endregion #region Encoding support (PEP 263) diff --git a/src/LanguageServer/Impl/Completion/PythonKeywords.cs b/src/Parsing/Impl/PythonKeywords.cs similarity index 97% rename from src/LanguageServer/Impl/Completion/PythonKeywords.cs rename to src/Parsing/Impl/PythonKeywords.cs index c0a8c29a1..f0e1b67e4 100644 --- a/src/LanguageServer/Impl/Completion/PythonKeywords.cs +++ b/src/Parsing/Impl/PythonKeywords.cs @@ -16,13 +16,12 @@ using System; using System.Collections.Generic; using System.Linq; -using Microsoft.Python.Parsing; -namespace Microsoft.Python.LanguageServer.Completion { +namespace Microsoft.Python.Parsing { /// /// The canonical source of keyword names for Python. /// - internal static class PythonKeywords { + public static class PythonKeywords { /// /// Returns true if the specified identifier is a keyword in a /// particular version of Python. diff --git a/src/Parsing/Impl/PythonLanguageVersion.cs b/src/Parsing/Impl/PythonLanguageVersion.cs index 5c3142c0f..62386af4f 100644 --- a/src/Parsing/Impl/PythonLanguageVersion.cs +++ b/src/Parsing/Impl/PythonLanguageVersion.cs @@ -37,7 +37,10 @@ public enum PythonLanguageVersion { V35 = 0x0305, V36 = 0x0306, V37 = 0x0307, - V38 = 0x0308 + V38 = 0x0308, + V39 = 0x0309, + V310 = 0x030a, + V311 = 0x030b, } public static class PythonLanguageVersionExtensions { diff --git a/src/Parsing/Impl/Resources.Designer.cs b/src/Parsing/Impl/Resources.Designer.cs index 507ec0381..07b9d5e34 100644 --- a/src/Parsing/Impl/Resources.Designer.cs +++ b/src/Parsing/Impl/Resources.Designer.cs @@ -466,7 +466,7 @@ internal static string NamedArgumentsMustFollowBareErrorMsg { } /// - /// Looks up a localized string similar to Cannot use named assignment with {0}. + /// Looks up a localized string similar to cannot use named assignment with {0}. /// internal static string NamedAssignmentWithErrorMsg { get { @@ -475,7 +475,7 @@ internal static string NamedAssignmentWithErrorMsg { } /// - /// Looks up a localized string similar to Named expression must be parenthesized in this context. + /// Looks up a localized string similar to named expression must be parenthesized in this context. /// internal static string NamedExpressionCtxtErrorMsg { get { @@ -483,6 +483,42 @@ internal static string NamedExpressionCtxtErrorMsg { } } + /// + /// Looks up a localized string similar to assignment expression within a comprehension cannot be used in a class body. + /// + internal static string NamedExpressionInClassBodyErrorMsg { + get { + return ResourceManager.GetString("NamedExpressionInClassBodyErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to assignment expression cannot be used in a comprehension iterable expression. + /// + internal static string NamedExpressionInComprehensionIteratorErrorMsg { + get { + return ResourceManager.GetString("NamedExpressionInComprehensionIteratorErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to comprehension inner loop cannot rebind assignment expression target '{0}'. + /// + internal static string NamedExpressionIteratorRebindsNamedErrorMsg { + get { + return ResourceManager.GetString("NamedExpressionIteratorRebindsNamedErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to assignment expression cannot rebind comprehension iteration variable '{0}'. + /// + internal static string NamedExpressionRebindIteratorErrorMsg { + get { + return ResourceManager.GetString("NamedExpressionRebindIteratorErrorMsg", resourceCulture); + } + } + /// /// Looks up a localized string similar to non-keyword arg after keyword arg. /// @@ -573,6 +609,60 @@ internal static string PositionalArgumentKeywardArgumentUnpackingErrorMsg { } } + /// + /// Looks up a localized string similar to positional only marker after ** args not allowed. + /// + internal static string PositionalOnlyMarkerAfterDictArgsErrorMsg { + get { + return ResourceManager.GetString("PositionalOnlyMarkerAfterDictArgsErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to positional only marker after * args not allowed. + /// + internal static string PositionalOnlyMarkerAfterListArgsErrorMsg { + get { + return ResourceManager.GetString("PositionalOnlyMarkerAfterListArgsErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to positional only marker may not have annotation. + /// + internal static string PositionalOnlyMarkerAnnotationErrorMsg { + get { + return ResourceManager.GetString("PositionalOnlyMarkerAnnotationErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to positional only marker may not have default. + /// + internal static string PositionalOnlyMarkerDefaultErrorMsg { + get { + return ResourceManager.GetString("PositionalOnlyMarkerDefaultErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to duplicate positional only marker. + /// + internal static string PositionalOnlyMarkerDuplicateErrorMsg { + get { + return ResourceManager.GetString("PositionalOnlyMarkerDuplicateErrorMsg", resourceCulture); + } + } + + /// + /// Looks up a localized string similar to positional only marker may not be first parameter. + /// + internal static string PositionalOnlyMarkerFirstParamErrorMessage { + get { + return ResourceManager.GetString("PositionalOnlyMarkerFirstParamErrorMessage", resourceCulture); + } + } + /// /// Looks up a localized string similar to positional parameter after * args not allowed. /// diff --git a/src/Parsing/Impl/Resources.resx b/src/Parsing/Impl/Resources.resx index f7b4ea964..34306da0a 100644 --- a/src/Parsing/Impl/Resources.resx +++ b/src/Parsing/Impl/Resources.resx @@ -253,10 +253,22 @@ named arguments must follow bare * - Cannot use named assignment with {0} + cannot use named assignment with {0} - Named expression must be parenthesized in this context + named expression must be parenthesized in this context + + + assignment expression within a comprehension cannot be used in a class body + + + assignment expression cannot be used in a comprehension iterable expression + + + comprehension inner loop cannot rebind assignment expression target '{0}' + + + assignment expression cannot rebind comprehension iteration variable '{0}' non-keyword arg after keyword arg @@ -288,6 +300,24 @@ positional argument follows keyword argument unpacking + + positional only marker after ** args not allowed + + + positional only marker after * args not allowed + + + positional only marker may not have annotation + + + positional only marker may not have default + + + duplicate positional only marker + + + positional only marker may not be first parameter + positional parameter after * args not allowed diff --git a/src/Parsing/Impl/Tokens/Tokenizer.cs b/src/Parsing/Impl/Tokens/Tokenizer.cs index ec5622252..5adca6489 100644 --- a/src/Parsing/Impl/Tokens/Tokenizer.cs +++ b/src/Parsing/Impl/Tokens/Tokenizer.cs @@ -111,6 +111,16 @@ public List ReadTokens(int characterCount) { return tokens; } + /// + /// Current state, to be passed to + /// or ./> + /// + /// + /// This is used by PTVS to initialize a tokenizer with an existing state. + /// Do not remove, even though it is unused within language server. + /// + public object CurrentState => _state; + public int CurrentLine => _newLineLocations.Count; public SourceLocation CurrentPosition => IndexToLocation(CurrentIndex); @@ -1812,8 +1822,14 @@ private Token NextOperator(int ch) { case '~': return Tokens.TwiddleToken; case '@': - if (LanguageVersion >= PythonLanguageVersion.V35 && NextChar('=')) { - return Tokens.MatMultiplyEqualToken; + if (LanguageVersion >= PythonLanguageVersion.V35) { + if (NextChar('=')) { + return Tokens.MatMultiplyEqualToken; + } + if (GroupingLevel > 0) { + // @ can't be a decorator here. + return Tokens.MatMultiplyToken; + } } return Tokens.AtToken; } diff --git a/src/Parsing/Test/Microsoft.Python.Parsing.Tests.csproj b/src/Parsing/Test/Microsoft.Python.Parsing.Tests.csproj index b0c838cfb..578db784e 100644 --- a/src/Parsing/Test/Microsoft.Python.Parsing.Tests.csproj +++ b/src/Parsing/Test/Microsoft.Python.Parsing.Tests.csproj @@ -1,6 +1,6 @@  - netcoreapp3.0 + netcoreapp3.1 Microsoft.Python.Parsing.Tests Microsoft.Python.Parsing.Tests @@ -27,10 +27,10 @@ - - - - + + + + all runtime; build; native; contentfiles; analyzers diff --git a/src/Parsing/Test/ParserRoundTripTest.cs b/src/Parsing/Test/ParserRoundTripTest.cs index 68359bcf0..dd7041f6a 100644 --- a/src/Parsing/Test/ParserRoundTripTest.cs +++ b/src/Parsing/Test/ParserRoundTripTest.cs @@ -1,4 +1,4 @@ -// Python Tools for Visual Studio +// Python Tools for Visual Studio // Copyright(c) Microsoft Corporation // All rights reserved. // @@ -1566,6 +1566,11 @@ public void RoundTripInvalidClassName() { TestOneString(PythonLanguageVersion.V35, "\nclass {{ invalid_name }}MyClass:\n"); } + [TestMethod, Priority(0)] + public void RoundTripPositionalOnly() { + TestOneString(PythonLanguageVersion.V38, "def f(a, b, /, c, d, *, e, f): ..."); + } + private static void RoundTripStdLibTest(InterpreterConfiguration configuration) { configuration.AssertInstalled(); @@ -1579,7 +1584,7 @@ private static void RoundTripStdLibTest(InterpreterConfiguration configuration) ImmutableArray.Empty); var pathResolverSnapshot = pathResolver.CurrentSnapshot; - var modules = pathResolverSnapshot.GetAllModuleNames() + var modules = pathResolverSnapshot.GetAllImportableModuleNames() .Select(n => pathResolverSnapshot.GetModuleImportFromModuleName(n)) .Where(i => i.RootPath.PathEquals(configuration.SitePackagesPath)) .ToList(); @@ -1602,16 +1607,19 @@ private static void RoundTripStdLibTest(InterpreterConfiguration configuration) } [TestMethod, Priority(0)] - public void RoundTripStdLib27() => RoundTripStdLibTest(PythonVersions.Python27 ?? PythonVersions.Python27_x64); + public void RoundTripStdLib27() => RoundTripStdLibTest(PythonVersions.Required_Python27X); + + [TestMethod, Priority(0)] + public void RoundTripStdLib35() => RoundTripStdLibTest(PythonVersions.Required_Python35X); [TestMethod, Priority(0)] - public void RoundTripStdLib35() => RoundTripStdLibTest(PythonVersions.Python35 ?? PythonVersions.Python35_x64); + public void RoundTripStdLib36() => RoundTripStdLibTest(PythonVersions.Required_Python36X); [TestMethod, Priority(0)] - public void RoundTripStdLib36() => RoundTripStdLibTest(PythonVersions.Python36 ?? PythonVersions.Python36_x64); + public void RoundTripStdLib37() => RoundTripStdLibTest(PythonVersions.Required_Python37X); [TestMethod, Priority(0)] - public void RoundTripStdLib37() => RoundTripStdLibTest(PythonVersions.Python37 ?? PythonVersions.Python37_x64); + public void RoundTripStdLib38() => RoundTripStdLibTest(PythonVersions.Required_Python38X); [TestMethod, Priority(0)] public void GroupingRecovery() { diff --git a/src/Parsing/Test/ParserTests.cs b/src/Parsing/Test/ParserTests.cs index 998ffd662..ff1f08a3b 100644 --- a/src/Parsing/Test/ParserTests.cs +++ b/src/Parsing/Test/ParserTests.cs @@ -57,6 +57,7 @@ public class ParserTests { internal static readonly PythonLanguageVersion[] V33AndV34 = AllVersions.Where(v => v >= PythonLanguageVersion.V33 && v <= PythonLanguageVersion.V34).ToArray(); internal static readonly PythonLanguageVersion[] V33AndUp = AllVersions.Where(v => v >= PythonLanguageVersion.V33).ToArray(); internal static readonly PythonLanguageVersion[] V35AndUp = AllVersions.Where(v => v >= PythonLanguageVersion.V35).ToArray(); + internal static readonly PythonLanguageVersion[] V35_V37Versions = AllVersions.Where(v => v >= PythonLanguageVersion.V35 && v <= PythonLanguageVersion.V37).ToArray(); internal static readonly PythonLanguageVersion[] V36AndUp = AllVersions.Where(v => v >= PythonLanguageVersion.V36).ToArray(); internal static readonly PythonLanguageVersion[] V37AndUp = AllVersions.Where(v => v >= PythonLanguageVersion.V37).ToArray(); internal static readonly PythonLanguageVersion[] V38AndUp = AllVersions.Where(v => v >= PythonLanguageVersion.V38).ToArray(); @@ -115,7 +116,7 @@ public void FStringErrors() { new ErrorResult("f-string: expecting '}'", new SourceSpan(19, 4, 19, 5)), new ErrorResult("unexpected token 'import'", new SourceSpan(21, 4, 21, 10)), new ErrorResult("f-string: empty expression not allowed", new SourceSpan(23, 4, 23, 5)), - new ErrorResult("unexpected token '='", new SourceSpan(25, 6, 25, 7)), + new ErrorResult("f-string: expecting '}' but found '='", new SourceSpan(25, 6, 25, 7)), new ErrorResult("expected ':'", new SourceSpan(27, 12, 27, 12)), new ErrorResult("f-string: lambda must be inside parentheses", new SourceSpan(27, 4, 27, 12)), new ErrorResult("f-string: expecting '}'", new SourceSpan(29, 6, 29, 7)), @@ -124,7 +125,9 @@ public void FStringErrors() { new ErrorResult("f-string: unmatched ')'", new SourceSpan(35, 4, 35, 5)), new ErrorResult("f-string: unmatched ')'", new SourceSpan(37, 6, 37, 7)), new ErrorResult("f-string: closing parenthesis '}' does not match opening parenthesis '('", new SourceSpan(39, 6, 39, 7)), - new ErrorResult("f-string: unmatched ']'", new SourceSpan(41, 4, 41, 5)) + new ErrorResult("f-string: unmatched ']'", new SourceSpan(41, 4, 41, 5)), + new ErrorResult("unexpected EOF while parsing", new SourceSpan(43, 7, 43, 7)), + new ErrorResult("f-string: expecting '}' but found '='", new SourceSpan(43, 7, 43, 8)) ); } @@ -407,6 +410,72 @@ public void FStrings() { ) ) ) + ), + CheckExprStmt( + CheckFString( + CheckFormattedValue( + CheckConditionalExpression( + CheckNameExpr("x"), + CheckBinaryExpression(CheckNameExpr("x"), PythonOperator.Equal, CheckConstant(1.0)), + CheckConstant(0) + ) + ) + ) + ), + CheckExprStmt( + CheckFString( + CheckFormattedValue( + CheckConditionalExpression( + CheckNameExpr("x"), + CheckBinaryExpression(CheckNameExpr("x"), PythonOperator.NotEqual, CheckConstant(1.0)), + CheckConstant(0) + ) + ) + ) + ), + CheckExprStmt( + CheckFString( + CheckFormattedValue( + CheckConditionalExpression( + CheckNameExpr("x"), + CheckBinaryExpression(CheckNameExpr("x"), PythonOperator.GreaterThanOrEqual, CheckConstant(1.0)), + CheckConstant(0) + ) + ) + ) + ), + CheckExprStmt( + CheckFString( + CheckFormattedValue( + CheckConditionalExpression( + CheckNameExpr("x"), + CheckBinaryExpression(CheckNameExpr("x"), PythonOperator.LessThanOrEqual, CheckConstant(1.0)), + CheckConstant(0) + ) + ) + ) + ), + CheckExprStmt( + CheckFString( + CheckFormattedValue( + CheckConditionalExpression( + CheckNameExpr("x"), + CheckBinaryExpression(CheckNameExpr("x"), PythonOperator.GreaterThan, CheckConstant(1.0)), + CheckConstant(0) + ) + ) + ) + ), + CheckExprStmt( + CheckFString( + CheckFormattedValue( + CheckConditionalExpression( + CheckNameExpr("x"), + CheckBinaryExpression(CheckNameExpr("x"), PythonOperator.LessThan, CheckConstant(1.0)), + CheckConstant(0) + ) + ) + ) ) ) ); @@ -477,7 +546,9 @@ public void FStringEqualsErrors() { ParseErrors("FStringEqualsErrors.py", PythonLanguageVersion.V38, new ErrorResult("f-string: expecting '}' but found 'f'", new SourceSpan(1, 9, 1, 10)), - new ErrorResult("f-string: expecting '}' but found 'a'", new SourceSpan(2, 10, 2, 11)) + new ErrorResult("f-string: expecting '}' but found 'a'", new SourceSpan(2, 10, 2, 11)), + new ErrorResult("unexpected EOF while parsing", new SourceSpan(3, 9, 3, 9)), + new ErrorResult("f-string: expecting '}' but found 'a'", new SourceSpan(3, 11, 3, 12)) ); } @@ -955,7 +1026,7 @@ public void Errors(int index, int line, int column) { ); } - foreach (var version in V35AndUp) { + foreach (var version in V35_V37Versions) { ParseErrorsWithOffset("AllErrors.py", version, initLoc, @@ -1015,6 +1086,66 @@ public void Errors(int index, int line, int column) { new ErrorResult("invalid syntax", new SourceSpan(161, 10, 161, 13)) ); } + + foreach (var version in V38AndUp) { + ParseErrorsWithOffset("AllErrors.py", + version, + initLoc, + new ErrorResult("future statement does not support import *", new SourceSpan(1, 1, 1, 25)), + new ErrorResult("future feature is not defined: *", new SourceSpan(1, 1, 1, 25)), + new ErrorResult("not a chance", new SourceSpan(2, 1, 2, 30)), + new ErrorResult("future feature is not defined: unknown", new SourceSpan(3, 1, 3, 31)), + new ErrorResult("default value must be specified here", new SourceSpan(5, 16, 5, 17)), + new ErrorResult("positional argument follows keyword argument", new SourceSpan(8, 12, 8, 13)), + new ErrorResult("unexpected token 'pass'", new SourceSpan(14, 1, 14, 5)), + new ErrorResult("sublist parameters are not supported in 3.x", new SourceSpan(17, 10, 17, 17)), + new ErrorResult("invalid parameter", new SourceSpan(20, 10, 20, 12)), + new ErrorResult("'break' outside loop", new SourceSpan(25, 1, 25, 6)), + new ErrorResult("'continue' not properly in loop", new SourceSpan(26, 1, 26, 9)), + new ErrorResult("expected expression after del", new SourceSpan(36, 1, 36, 4)), + new ErrorResult("can't delete binary operator", new SourceSpan(37, 5, 37, 8)), + new ErrorResult("can't delete unary operator", new SourceSpan(38, 5, 38, 7)), + new ErrorResult("can't delete or expression", new SourceSpan(39, 5, 39, 13)), + new ErrorResult("can't delete and expression", new SourceSpan(40, 5, 40, 14)), + new ErrorResult("can't delete dictionary display", new SourceSpan(41, 5, 41, 7)), + new ErrorResult("can't delete literal", new SourceSpan(42, 5, 42, 9)), + new ErrorResult("can't delete literal", new SourceSpan(43, 5, 43, 9)), + new ErrorResult("can't assign to literal", new SourceSpan(45, 1, 45, 5)), + new ErrorResult("can't assign to literal", new SourceSpan(46, 1, 46, 5)), + new ErrorResult("'return' outside function", new SourceSpan(48, 1, 48, 7)), + new ErrorResult("misplaced yield", new SourceSpan(55, 1, 55, 6)), + new ErrorResult("two starred expressions in assignment", new SourceSpan(68, 8, 68, 10)), + new ErrorResult("illegal expression for augmented assignment", new SourceSpan(70, 1, 70, 3)), + new ErrorResult("missing module name", new SourceSpan(72, 6, 72, 12)), + new ErrorResult("import * only allowed at module level", new SourceSpan(75, 19, 75, 20)), + new ErrorResult("from __future__ imports must occur at the beginning of the file", new SourceSpan(78, 1, 78, 32)), + new ErrorResult("nonlocal declaration not allowed at module level", new SourceSpan(82, 1, 82, 9)), + new ErrorResult("invalid syntax, only exception value is allowed in 3.x.", new SourceSpan(83, 10, 83, 15)), + new ErrorResult("default value must be specified here", new SourceSpan(99, 15, 99, 16)), + new ErrorResult("duplicate * args arguments", new SourceSpan(105, 13, 105, 15)), + new ErrorResult("duplicate * args arguments", new SourceSpan(108, 13, 108, 15)), + new ErrorResult("named arguments must follow bare *", new SourceSpan(111, 10, 111, 11)), + new ErrorResult("sublist parameters are not supported in 3.x", new SourceSpan(114, 10, 114, 16)), + new ErrorResult("sublist parameters are not supported in 3.x", new SourceSpan(117, 10, 117, 17)), + new ErrorResult("duplicate argument 'abc' in function definition", new SourceSpan(120, 12, 120, 15)), + new ErrorResult("sublist parameters are not supported in 3.x", new SourceSpan(123, 10, 123, 20)), + new ErrorResult("invalid parameter", new SourceSpan(127, 7, 127, 9)), + new ErrorResult("\", variable\" not allowed in 3.x - use \"as variable\" instead.", new SourceSpan(134, 17, 134, 20)), + new ErrorResult("default 'except' must be last", new SourceSpan(132, 1, 132, 8)), + new ErrorResult("\", variable\" not allowed in 3.x - use \"as variable\" instead.", new SourceSpan(144, 17, 144, 20)), + new ErrorResult("cannot mix bytes and nonbytes literals", new SourceSpan(147, 8, 147, 13)), + new ErrorResult("cannot mix bytes and nonbytes literals", new SourceSpan(148, 7, 148, 13)), + new ErrorResult("invalid syntax", new SourceSpan(149, 7, 149, 9)), + new ErrorResult("invalid syntax", new SourceSpan(149, 7, 149, 9)), + new ErrorResult("invalid syntax", new SourceSpan(150, 8, 150, 10)), + new ErrorResult("invalid syntax", new SourceSpan(150, 8, 150, 10)), + new ErrorResult("invalid syntax", new SourceSpan(152, 4, 152, 6)), + new ErrorResult("expected name", new SourceSpan(154, 3, 154, 5)), + new ErrorResult("invalid parameter", new SourceSpan(156, 7, 156, 13)), + new ErrorResult("invalid syntax", new SourceSpan(160, 12, 160, 13)), + new ErrorResult("invalid syntax", new SourceSpan(161, 10, 161, 13)) + ); + } } private void ParseErrorsWithOffset(string filename, PythonLanguageVersion version, SourceLocation initialLocation, @@ -1489,6 +1620,24 @@ public void MatMulOperator() { } } + [TestMethod, Priority(0)] + public void MatMulMultilineOperator() { + foreach (var version in V35AndUp) { + CheckAst( + ParseFile("MatMulOperator2.py", ErrorSink.Null, version), + CheckSuite( + CheckAssignment(Fob, CheckParenExpr(CheckBinaryExpression(One, PythonOperator.MatMultiply, Two))) + ) + ); + } + + foreach (var version in V3Versions.Except(V35AndUp)) { + ParseErrors("MatMulOperator2.py", version, new[] { + new ErrorResult("unexpected token ''", new SourceSpan(2, 6, 3, 1)) + }); + } + } + [TestMethod, Priority(0)] public void GroupingRecovery() { foreach (var version in AllVersions) { @@ -2748,6 +2897,82 @@ public void FuncDefTrailingComma() { } } + [TestMethod, Priority(0)] + public void PositionalOnly() { + foreach (var version in V38AndUp) { + var errors = new CollectingErrorSink(); + CheckAst( + ParseFile("PositionalOnly.py", errors, version), + CheckSuite( + CheckFuncDef("f", new[] { + CheckParameter("a", ParameterKind.PositionalOnly), + CheckParameter("b", ParameterKind.PositionalOnly), + CheckParameter(null, ParameterKind.PositionalOnlyMarker), + CheckParameter("c", ParameterKind.Normal), + CheckParameter("d", ParameterKind.Normal), + CheckParameter(null, ParameterKind.List), + CheckParameter("e", ParameterKind.KeywordOnly), + CheckParameter("f", ParameterKind.KeywordOnly), + }, CheckSuite(Pass)), + CheckFuncDef("pow2", new[] { + CheckParameter("x", ParameterKind.PositionalOnly), + CheckParameter("y", ParameterKind.PositionalOnly), + CheckParameter("z", ParameterKind.PositionalOnly, None), + CheckParameter(null, ParameterKind.PositionalOnlyMarker), + }, CheckSuite(Pass)), + CheckFuncDef("foo", new[] { + CheckParameter("name", ParameterKind.PositionalOnly), + CheckParameter(null, ParameterKind.PositionalOnlyMarker), + CheckParameter("kwds", ParameterKind.Dictionary), + }, CheckSuite(Pass)) + ) + ); + errors.Errors.Should().BeEmpty(); + } + } + + [TestMethod, Priority(0)] + public void PositionalOnlyErrors() { + foreach (var version in V38AndUp) { + var errors = new CollectingErrorSink(); + ParseFile("PositionalOnlyErrors.py", errors, version); + errors.Errors.Should().BeEquivalentTo(new[] { + new ErrorResult("positional only marker after * args not allowed", new SourceSpan(1, 14, 1, 15)), + new ErrorResult("positional only marker after * args not allowed", new SourceSpan(4, 16, 4, 17)), + new ErrorResult("positional only marker after * args not allowed", new SourceSpan(7, 26, 7, 27)), + new ErrorResult("positional only marker after ** args not allowed", new SourceSpan(10, 20, 10, 21)), + new ErrorResult("duplicate positional only marker", new SourceSpan(13, 16, 13, 17)), + new ErrorResult("positional only marker may not have annotation", new SourceSpan(16, 10, 16, 15)), + new ErrorResult("positional only marker may not have default", new SourceSpan(19, 10, 19, 16)), + new ErrorResult("positional only marker may not be first parameter", new SourceSpan(22, 7, 22, 8)), + }); + } + } + + [TestMethod, Priority(0)] + public void ForFinallyContinue() { + foreach (var version in V38AndUp) { + var errors = new CollectingErrorSink(); + CheckAst( + ParseFile("ForFinallyContinue.py", errors, version), + CheckSuite( + CheckForStmt( + CheckNameExpr("x"), + CheckNameExpr("l"), + CheckSuite( + CheckTryStmt( + CheckSuite(Pass), + Array.Empty< Action>(), + _finally: CheckSuite(Continue) + ) + ) + ) + ) + ); + errors.Errors.Should().BeEmpty(); + } + } + [TestMethod, Priority(0)] public void CoroutineDef() { foreach (var version in V35AndUp) { @@ -3012,6 +3237,49 @@ public void NamedExpressions() { CheckSuite(Pass) ) ) + ), + CheckClassDef( + "LambdaTop", + CheckSuite( + CheckExprStmt( + CheckListComp( + CheckParenExpr( + CheckLambda( + NoParameters, + CheckParenExpr( + CheckNamedExpr( + CheckNameExpr("z"), + CheckNameExpr("x") + ) + ) + ) + ), + CompFor( + CheckNameExpr("x"), + CheckCallExpression(CheckNameExpr("range"), PositionalArg(One)) + ) + ) + ) + ) + ), + CheckExprStmt( + CheckListComp( + CheckParenExpr( + CheckLambda( + new[] { CheckParameter("x") }, + CheckParenExpr( + CheckNamedExpr( + CheckNameExpr("x"), + CheckNameExpr("x") + ) + ) + ) + ), + CompFor( + CheckNameExpr("x"), + CheckCallExpression(CheckNameExpr("range"), PositionalArg(One)) + ) + ) ) ) ); @@ -3025,15 +3293,41 @@ public void NamedExpressionsErrors() { var errors = new CollectingErrorSink(); ParseFile("NamedExpressionsErrors.py", errors, version); errors.Errors.Should().BeEquivalentTo(new[] { - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(1, 3, 1, 5)), - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(2, 11, 2, 13)), - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(3, 7, 3, 9)), - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(4, 19, 4, 21)), - new ErrorResult("Cannot use named assignment with subscript", new SourceSpan(8, 2, 8, 6)), - new ErrorResult("Cannot use named assignment with attribute", new SourceSpan(9, 2, 9, 5)), - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(12, 9, 12, 11)), - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(14, 21, 14, 23)), - new ErrorResult("Named expression must be parenthesized in this context", new SourceSpan(17, 9, 17, 11)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(1, 3, 1, 5)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(2, 11, 2, 13)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(3, 7, 3, 9)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(4, 19, 4, 21)), + new ErrorResult("cannot use named assignment with subscript", new SourceSpan(8, 2, 8, 6)), + new ErrorResult("cannot use named assignment with attribute", new SourceSpan(9, 2, 9, 5)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(12, 9, 12, 11)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(14, 21, 14, 23)), + new ErrorResult("named expression must be parenthesized in this context", new SourceSpan(17, 9, 17, 11)), + }); + } + } + + [TestMethod, Priority(0)] + public void NamedExpressionScopeErrors() { + foreach (var version in V38AndUp) { + var errors = new CollectingErrorSink(); + ParseFile("NamedExpressionScopeErrors.py", errors, version); + errors.Errors.Should().BeEquivalentTo(new[] { + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(1, 17, 1, 32)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(2, 27, 2, 42)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(3, 20, 3, 35)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(4, 17, 4, 32)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(5, 33, 5, 48)), + new ErrorResult("assignment expression cannot rebind comprehension iteration variable 'j'", new SourceSpan(7, 4, 7, 5)), + new ErrorResult("assignment expression cannot rebind comprehension iteration variable 'i'", new SourceSpan(8, 2, 8, 3)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(9, 16, 9, 26)), + new ErrorResult("assignment expression cannot rebind comprehension iteration variable 'i'", new SourceSpan(11, 13, 11, 14)), + new ErrorResult("assignment expression cannot rebind comprehension iteration variable 'j'", new SourceSpan(12, 34, 12, 35)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(14, 16, 14, 26)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(15, 34, 15, 44)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(16, 28, 16, 38)), + new ErrorResult("assignment expression cannot be used in a comprehension iterable expression", new SourceSpan(17, 25, 17, 35)), + new ErrorResult("assignment expression within a comprehension cannot be used in a class body", new SourceSpan(20, 7, 20, 13)), + new ErrorResult("comprehension inner loop cannot rebind assignment expression target 'j'", new SourceSpan(22, 43, 22, 44)), }); } } @@ -3901,9 +4195,13 @@ private static Action CheckTryStmt(Action body, Action CheckDelStmt(params Action[] deletes) { }; } - private Action CheckParenExpr(Action value) { + private static Action CheckParenExpr(Action value) { return expr => { Assert.AreEqual(typeof(ParenthesisExpression), expr.GetType()); var paren = (ParenthesisExpression)expr; diff --git a/src/Parsing/Test/PythonVersions.cs b/src/Parsing/Test/PythonVersions.cs index a199eba79..9efdc52cc 100644 --- a/src/Parsing/Test/PythonVersions.cs +++ b/src/Parsing/Test/PythonVersions.cs @@ -26,11 +26,13 @@ public static class PythonVersions { public static readonly InterpreterConfiguration Python36 = GetCPythonVersion(PythonLanguageVersion.V36, InterpreterArchitecture.x86); public static readonly InterpreterConfiguration Python37 = GetCPythonVersion(PythonLanguageVersion.V37, InterpreterArchitecture.x86); public static readonly InterpreterConfiguration Python38 = GetCPythonVersion(PythonLanguageVersion.V38, InterpreterArchitecture.x86); + public static readonly InterpreterConfiguration Python39 = GetCPythonVersion(PythonLanguageVersion.V39, InterpreterArchitecture.x86); public static readonly InterpreterConfiguration Python27_x64 = GetCPythonVersion(PythonLanguageVersion.V27, InterpreterArchitecture.x64); public static readonly InterpreterConfiguration Python35_x64 = GetCPythonVersion(PythonLanguageVersion.V35, InterpreterArchitecture.x64); public static readonly InterpreterConfiguration Python36_x64 = GetCPythonVersion(PythonLanguageVersion.V36, InterpreterArchitecture.x64); public static readonly InterpreterConfiguration Python37_x64 = GetCPythonVersion(PythonLanguageVersion.V37, InterpreterArchitecture.x64); public static readonly InterpreterConfiguration Python38_x64 = GetCPythonVersion(PythonLanguageVersion.V38, InterpreterArchitecture.x64); + public static readonly InterpreterConfiguration Python39_x64 = GetCPythonVersion(PythonLanguageVersion.V39, InterpreterArchitecture.x64); public static readonly InterpreterConfiguration Anaconda27 = GetAnacondaVersion(PythonLanguageVersion.V27, InterpreterArchitecture.x86); public static readonly InterpreterConfiguration Anaconda27_x64 = GetAnacondaVersion(PythonLanguageVersion.V27, InterpreterArchitecture.x64); public static readonly InterpreterConfiguration Anaconda36 = GetAnacondaVersion(PythonLanguageVersion.V36, InterpreterArchitecture.x86); @@ -63,6 +65,7 @@ public static class PythonVersions { public static InterpreterConfiguration Required_Python36X => Python36 ?? Python36_x64 ?? NotInstalled("v3.6"); public static InterpreterConfiguration Required_Python37X => Python37 ?? Python37_x64 ?? NotInstalled("v3.7"); public static InterpreterConfiguration Required_Python38X => Python38 ?? Python38_x64 ?? NotInstalled("v3.8"); + public static InterpreterConfiguration Required_Python39X => Python39 ?? Python39_x64 ?? NotInstalled("v3.9"); public static InterpreterConfiguration LatestAvailable => LatestAvailable3X ?? LatestAvailable2X; diff --git a/src/Publish/SignLayout.csproj b/src/Publish/SignLayout.csproj index 37a0a6a4a..d582ef769 100644 --- a/src/Publish/SignLayout.csproj +++ b/src/Publish/SignLayout.csproj @@ -8,7 +8,7 @@ - false + false false false @@ -18,7 +18,7 @@ all runtime; build; native; contentfiles; analyzers - + @@ -26,11 +26,9 @@ - + - - @@ -40,6 +38,7 @@ + @@ -50,6 +49,7 @@ NuGet Microsoft400 + StrongName diff --git a/src/Publish/global.json b/src/Publish/global.json index 9ec93e70e..48f7edbb0 100644 --- a/src/Publish/global.json +++ b/src/Publish/global.json @@ -1,5 +1,5 @@ { "sdk": { - "version": "3.0.100-rc1-014190" + "version": "3.1.409" } } \ No newline at end of file diff --git a/src/UnitTests/Core/Impl/MSTest/PermutationalTestMethodAttribute.cs b/src/UnitTests/Core/Impl/MSTest/PermutationDataRowAttribute.cs similarity index 60% rename from src/UnitTests/Core/Impl/MSTest/PermutationalTestMethodAttribute.cs rename to src/UnitTests/Core/Impl/MSTest/PermutationDataRowAttribute.cs index 524c7da90..c78f64454 100644 --- a/src/UnitTests/Core/Impl/MSTest/PermutationalTestMethodAttribute.cs +++ b/src/UnitTests/Core/Impl/MSTest/PermutationDataRowAttribute.cs @@ -1,4 +1,4 @@ -// Python Tools for Visual Studio +// Python Tools for Visual Studio // Copyright(c) Microsoft Corporation // All rights reserved. // @@ -21,34 +21,46 @@ using Microsoft.VisualStudio.TestTools.UnitTesting; namespace Microsoft.Python.UnitTests.Core.MSTest { - [AttributeUsage(AttributeTargets.Method)] - public class PermutationalTestMethodAttribute : DataTestMethodAttribute, ITestDataSource { - private readonly int _count; - private readonly int[] _fixedPermutation; + [AttributeUsage(AttributeTargets.Method, AllowMultiple = true)] + public class PermutationDataRowAttribute : Attribute, ITestDataSource { + private readonly object[] _source; - public string NameFormat { get; set; } = "{0} ({1})"; + public PermutationDataRowAttribute() { + _source = new object[0]; + } - public PermutationalTestMethodAttribute(int count, params int[] fixedPermutation) { - _count = count; - _fixedPermutation = fixedPermutation; + public PermutationDataRowAttribute(object data) { + _source = new object[1] {data}; } - public IEnumerable GetData(MethodInfo methodInfo) { - if (_fixedPermutation != null && _fixedPermutation.Length > 0) { - yield return new object[] { _fixedPermutation }; - yield break; + public PermutationDataRowAttribute(object data, params object[] moreData) { + if (moreData == null) { + moreData = new object[1]; } - var permutationsIndexes = GetPermutationIndexes(_count); - foreach (var permutationIndexes in permutationsIndexes) { - yield return new object []{ permutationIndexes }; - } + _source = new object[moreData.Length + 1]; + _source[0] = data; + Array.Copy(moreData, 0, _source, 1, moreData.Length); } - public string GetDisplayName(MethodInfo methodInfo, object[] data) { - var names = string.Join(", ", (int[])data[0]); - return string.Format(CultureInfo.InvariantCulture, NameFormat, methodInfo.Name, names); + public IEnumerable GetData(MethodInfo methodInfo) { + var permutationsIndexes = GetPermutationIndexes(_source.Length); + var data = new object[permutationsIndexes.Length][]; + for (var dataIndex = 0; dataIndex < permutationsIndexes.Length; dataIndex++) { + var permutationIndexes = permutationsIndexes[dataIndex]; + var dataRow = new object[_source.Length]; + for (var i = 0; i < dataRow.Length; i++) { + dataRow[i] = _source[permutationIndexes[i]]; + } + + data[dataIndex] = dataRow; + } + + return data; } + + public string GetDisplayName(MethodInfo methodInfo, object[] data) + => data == null ? null : string.Format(CultureInfo.InvariantCulture, "{0} ({1})", methodInfo.Name, string.Join(", ", data)); private int[][] GetPermutationIndexes(int count) { if (count == 0) { diff --git a/src/UnitTests/Core/Impl/MarkupUtils.cs b/src/UnitTests/Core/Impl/MarkupUtils.cs new file mode 100644 index 000000000..42feee415 --- /dev/null +++ b/src/UnitTests/Core/Impl/MarkupUtils.cs @@ -0,0 +1,244 @@ +// Python Tools for Visual Studio +// Copyright(c) Microsoft Corporation +// All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the License); you may not use +// this file except in compliance with the License. You may obtain a copy of the +// License at http://www.apache.org/licenses/LICENSE-2.0 +// +// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS +// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY +// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, +// MERCHANTABILITY OR NON-INFRINGEMENT. +// +// See the Apache Version 2.0 License for specific language governing +// permissions and limitations under the License. + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Text.RegularExpressions; +using Microsoft.Python.Core.Text; + +namespace Microsoft.Python.UnitTests.Core { + /// + /// To aid with testing, we define a special type of text file that can encode additional + /// information in it. This prevents a test writer from having to carry around multiple sources + /// of information that must be reconstituted. For example, instead of having to keep around the + /// contents of a file *and* and the location of the cursor, the tester can just provide a + /// string with the "$" character in it. This allows for easy creation of "FIT" tests where all + /// that needs to be provided are strings that encode every bit of state necessary in the string + /// itself. + /// + /// The current set of encoded features we support are: + /// + /// $$ - The position in the file. There can be at most one of these. + /// + /// [| ... |] - A span of text in the file. There can be many of these and they can be nested + /// and/or overlap the $ position. + /// + /// {|Name: ... |} A span of text in the file annotated with an identifier. There can be many of + /// these, including ones with the same name. + /// + /// Additional encoded features can be added on a case by case basis. + /// + public static class MarkupUtils { + private const string PositionString = "$$"; + private const string SpanStartString = "[|"; + private const string SpanEndString = "|]"; + private const string NamedSpanStartString = "{|"; + private const string NamedSpanEndString = "|}"; + + private static readonly Regex s_namedSpanStartRegex = new Regex(@"\{\| ([-_.A-Za-z0-9\+]+) \:", + RegexOptions.Multiline | RegexOptions.IgnorePatternWhitespace); + + private static void Parse( + string input, out string output, out int? position, out IDictionary> spans) { + position = null; + var tempSpans = new Dictionary>(); + + var outputBuilder = new StringBuilder(); + + var currentIndexInInput = 0; + var inputOutputOffset = 0; + + // A stack of span starts along with their associated annotation name. [||] spans simply + // have empty string for their annotation name. + var spanStartStack = new Stack>(); + + while (true) { + var matches = new List<(int index, string value)>(); + AddMatch(input, PositionString, currentIndexInInput, matches); + AddMatch(input, SpanStartString, currentIndexInInput, matches); + AddMatch(input, SpanEndString, currentIndexInInput, matches); + AddMatch(input, NamedSpanEndString, currentIndexInInput, matches); + + var namedSpanStartMatch = s_namedSpanStartRegex.Match(input, currentIndexInInput); + if (namedSpanStartMatch.Success) { + matches.Add((namedSpanStartMatch.Index, namedSpanStartMatch.Value)); + } + + if (matches.Count == 0) { + // No more markup to process. + break; + } + + var orderedMatches = matches.OrderBy(t => t.index).ToList(); + if (orderedMatches.Count >= 2 && + spanStartStack.Count > 0 && + matches[0].Item1 == matches[1].Item1 - 1) { + // We have a slight ambiguity with cases like these: + // + // [|] [|} + // + // Is it starting a new match, or ending an existing match. As a workaround, we + // special case these and consider it ending a match if we have something on the + // stack already. + if ((matches[0].Item2 == SpanStartString && matches[1].Item2 == SpanEndString && spanStartStack.Peek().Item2 == string.Empty) || + (matches[0].Item2 == SpanStartString && matches[1].Item2 == NamedSpanEndString && spanStartStack.Peek().Item2 != string.Empty)) { + orderedMatches.RemoveAt(0); + } + } + + // Order the matches by their index + var firstMatch = orderedMatches.First(); + + var matchIndexInInput = firstMatch.Item1; + var matchString = firstMatch.Item2; + + var matchIndexInOutput = matchIndexInInput - inputOutputOffset; + outputBuilder.Append(input.Substring(currentIndexInInput, matchIndexInInput - currentIndexInInput)); + + currentIndexInInput = matchIndexInInput + matchString.Length; + inputOutputOffset += matchString.Length; + + switch (matchString.Substring(0, 2)) { + case PositionString: + if (position.HasValue) { + throw new ArgumentException(string.Format("Saw multiple occurrences of {0}", PositionString)); + } + + position = matchIndexInOutput; + break; + + case SpanStartString: + spanStartStack.Push(Tuple.Create(matchIndexInOutput, string.Empty)); + break; + + case SpanEndString: + if (spanStartStack.Count == 0) { + throw new ArgumentException(string.Format("Saw {0} without matching {1}", SpanEndString, SpanStartString)); + } + + if (spanStartStack.Peek().Item2.Length > 0) { + throw new ArgumentException(string.Format("Saw {0} without matching {1}", NamedSpanStartString, NamedSpanEndString)); + } + + PopSpan(spanStartStack, tempSpans, matchIndexInOutput); + break; + + case NamedSpanStartString: + var name = namedSpanStartMatch.Groups[1].Value; + spanStartStack.Push(Tuple.Create(matchIndexInOutput, name)); + break; + + case NamedSpanEndString: + if (spanStartStack.Count == 0) { + throw new ArgumentException(string.Format("Saw {0} without matching {1}", NamedSpanEndString, NamedSpanStartString)); + } + + if (spanStartStack.Peek().Item2.Length == 0) { + throw new ArgumentException(string.Format("Saw {0} without matching {1}", SpanStartString, SpanEndString)); + } + + PopSpan(spanStartStack, tempSpans, matchIndexInOutput); + break; + + default: + throw new InvalidOperationException(); + } + } + + if (spanStartStack.Count > 0) { + throw new ArgumentException(string.Format("Saw {0} without matching {1}", SpanStartString, SpanEndString)); + } + + // Append the remainder of the string. + outputBuilder.Append(input.Substring(currentIndexInInput)); + output = outputBuilder.ToString(); + spans = tempSpans.ToDictionary(kvp => kvp.Key, kvp => kvp.Value); + } + + private static V GetOrAdd(IDictionary dictionary, K key, Func function) { + if (!dictionary.TryGetValue(key, out var value)) { + value = function(key); + dictionary.Add(key, value); + } + + return value; + } + + private static void PopSpan( + Stack> spanStartStack, + IDictionary> spans, + int finalIndex) { + var spanStartTuple = spanStartStack.Pop(); + + var span = IndexSpan.FromBounds(spanStartTuple.Item1, finalIndex); + GetOrAdd(spans, spanStartTuple.Item2, _ => new List()).Add(span); + } + + private static void AddMatch(string input, string value, int currentIndex, List<(int index, string value)> matches) { + var index = input.IndexOf(value, currentIndex, StringComparison.Ordinal); + if (index >= 0) { + matches.Add((index, value)); + } + } + + private static void GetPositionAndSpans( + string input, out string output, out int? cursorPositionOpt, out List spans) { + Parse(input, out output, out cursorPositionOpt, out var dictionary); + + spans = GetOrAdd(dictionary, string.Empty, _ => new List()); + } + + public static void GetPositionAndNamedSpans( + string input, out string output, out int? cursorPositionOpt, out IDictionary> spans) { + Parse(input, out output, out cursorPositionOpt, out spans); + } + + public static void GetNamedSpans(string input, out string output, out IDictionary> spans) + => GetPositionAndNamedSpans(input, out output, out var _, out spans); + + public static void GetPositionAndSpans(string input, out string output, out int cursorPosition, out List spans) { + GetPositionAndSpans(input, out output, out int? pos, out spans); + cursorPosition = pos.Value; + } + + public static void GetPosition(string input, out string output, out int? cursorPosition) + => GetPositionAndSpans(input, out output, out cursorPosition, out List _); + + public static void GetPosition(string input, out string output, out int cursorPosition) + => GetPositionAndSpans(input, out output, out cursorPosition, out var _); + + public static void GetPositionAndSpan(string input, out string output, out int? cursorPosition, out IndexSpan? textSpan) { + GetPositionAndSpans(input, out output, out cursorPosition, out List spans); + textSpan = spans.Count == 0 ? null : (IndexSpan?)spans.Single(); + } + + public static void GetPositionAndSpan(string input, out string output, out int cursorPosition, out IndexSpan textSpan) { + GetPositionAndSpans(input, out output, out cursorPosition, out var spans); + textSpan = spans.Single(); + } + + public static void GetSpans(string input, out string output, out List spans) { + GetPositionAndSpans(input, out output, out int? pos, out spans); + } + + public static void GetSpan(string input, out string output, out IndexSpan textSpan) { + GetSpans(input, out output, out List spans); + textSpan = spans.Single(); + } + } +} diff --git a/src/UnitTests/Core/Impl/UnitTests.Core.csproj b/src/UnitTests/Core/Impl/UnitTests.Core.csproj index b4b821830..44e9489ca 100644 --- a/src/UnitTests/Core/Impl/UnitTests.Core.csproj +++ b/src/UnitTests/Core/Impl/UnitTests.Core.csproj @@ -7,14 +7,17 @@ - + - + all runtime; build; native; contentfiles; analyzers + + + \ No newline at end of file diff --git a/src/UnitTests/TestData/AstAnalysis/Library1.py b/src/UnitTests/TestData/AstAnalysis/Library1.py new file mode 100644 index 000000000..ab08492ee --- /dev/null +++ b/src/UnitTests/TestData/AstAnalysis/Library1.py @@ -0,0 +1,14 @@ +class Foo(): + """This is Foo and the docs should be here""" + def __init__(self): + self.x = 1234 + self.y = "str" + + +def make_foo(): + """This is make_foo and the docs should be here""" + return Foo() + +def print_foo(f): + """This is print_foo and the docs should be here""" + print(f"{f.x}, {f.y}") diff --git a/src/UnitTests/TestData/Grammar/FStringEqualsErrors.py b/src/UnitTests/TestData/Grammar/FStringEqualsErrors.py index 821496991..61d2f74dd 100644 --- a/src/UnitTests/TestData/Grammar/FStringEqualsErrors.py +++ b/src/UnitTests/TestData/Grammar/FStringEqualsErrors.py @@ -1,2 +1,3 @@ f"{name=foo}" -f"{bad = a }" \ No newline at end of file +f"{bad = a }" +f"{bad *= a }" \ No newline at end of file diff --git a/src/UnitTests/TestData/Grammar/FStringErrors.py b/src/UnitTests/TestData/Grammar/FStringErrors.py index 796ab9b6c..a742d892b 100644 --- a/src/UnitTests/TestData/Grammar/FStringErrors.py +++ b/src/UnitTests/TestData/Grammar/FStringErrors.py @@ -39,3 +39,5 @@ f'{ (} }' f'{]}' + +f'{a /= 1}' diff --git a/src/UnitTests/TestData/Grammar/FStrings.py b/src/UnitTests/TestData/Grammar/FStrings.py index d2fd893c3..486d703f4 100644 --- a/src/UnitTests/TestData/Grammar/FStrings.py +++ b/src/UnitTests/TestData/Grammar/FStrings.py @@ -48,3 +48,10 @@ def f(): f'{(lambda x: x + 1)(1)}' f"{[1,2,1][1:]}" + +f'{x if x == 1.0 else 0}' +f'{x if x != 1.0 else 0}' +f'{x if x >= 1.0 else 0}' +f'{x if x <= 1.0 else 0}' +f'{x if x > 1.0 else 0}' +f'{x if x < 1.0 else 0}' diff --git a/src/UnitTests/TestData/Grammar/ForFinallyContinue.py b/src/UnitTests/TestData/Grammar/ForFinallyContinue.py new file mode 100644 index 000000000..cd08b9e40 --- /dev/null +++ b/src/UnitTests/TestData/Grammar/ForFinallyContinue.py @@ -0,0 +1,5 @@ +for x in l: + try: + pass + finally: + continue diff --git a/src/UnitTests/TestData/Grammar/MatMulOperator2.py b/src/UnitTests/TestData/Grammar/MatMulOperator2.py new file mode 100644 index 000000000..1da3e14a2 --- /dev/null +++ b/src/UnitTests/TestData/Grammar/MatMulOperator2.py @@ -0,0 +1,4 @@ +fob = ( + 1 + @ 2 +) diff --git a/src/UnitTests/TestData/Grammar/NamedExpressionScopeErrors.py b/src/UnitTests/TestData/Grammar/NamedExpressionScopeErrors.py new file mode 100644 index 000000000..922b63c5b --- /dev/null +++ b/src/UnitTests/TestData/Grammar/NamedExpressionScopeErrors.py @@ -0,0 +1,22 @@ +[x for x, y in (pairs2 := pairs) if x % 2 == 0] +[x for x, y in ([1, 2, 3, pairs2 := pairs]) if x % 2 == 0] +{x: y for x, y in (pairs2 := pairs) if x % 2 == 0} +{x for x, y in (pairs2 := pairs) if x % 2 == 0} +foo = (x for x, y in ([1, 2, 3, pairs2 := pairs]) if x % 2 == 0) + +[[(j := j) for i in range(5)] for j in range(5)] # INVALID +[i := 0 for i, j in stuff] # INVALID +[i+1 for i in (i := stuff)] # INVALID + +[False and (i := 0) for i, j in stuff] # INVALID +[i for i, j in stuff if True or (j := 1)] # INVALID + +[i+1 for i in (j := stuff)] # INVALID +[i+1 for i in range(2) for j in (k := stuff)] # INVALID +[i+1 for i in [j for j in (k := stuff)]] # INVALID +[i+1 for i in (lambda: (j := stuff))()] # INVALID + +class Example: + [(j := i) for i in range(5)] # INVALID + +[i for i in stuff if True or (j := 1) for j in range(10)] diff --git a/src/UnitTests/TestData/Grammar/NamedExpressions.py b/src/UnitTests/TestData/Grammar/NamedExpressions.py index f310c5b9d..da1a8c607 100644 --- a/src/UnitTests/TestData/Grammar/NamedExpressions.py +++ b/src/UnitTests/TestData/Grammar/NamedExpressions.py @@ -32,3 +32,8 @@ def foo(answer: (p := 42) = 5, cat = ''): if x := a: pass + +class LambdaTop: + [(lambda: (z := x)) for x in range(1)] + +[(lambda x: (x := x)) for x in range(1)] diff --git a/src/UnitTests/TestData/Grammar/PositionalOnly.py b/src/UnitTests/TestData/Grammar/PositionalOnly.py new file mode 100644 index 000000000..5a4018c1e --- /dev/null +++ b/src/UnitTests/TestData/Grammar/PositionalOnly.py @@ -0,0 +1,8 @@ +def f(a, b, /, c, d, *, e, f): + pass + +def pow2(x, y, z=None, /): + pass + +def foo(name, /, **kwds): + pass \ No newline at end of file diff --git a/src/UnitTests/TestData/Grammar/PositionalOnlyErrors.py b/src/UnitTests/TestData/Grammar/PositionalOnlyErrors.py new file mode 100644 index 000000000..7d9dca8dc --- /dev/null +++ b/src/UnitTests/TestData/Grammar/PositionalOnlyErrors.py @@ -0,0 +1,23 @@ +def f(a, *b, /): + pass + +def f(a, *, b, /): + pass + +def f(a, *, b, **kwargs, /): + pass + +def f(a, **kwargs, /): + pass + +def f(a, /, b, /, c): + pass + +def f(a, /:int): + pass + +def f(a, / =123): + pass + +def f(/): + pass