Member

killercup commented on Oct 13, 2015 •

Rendered

liigo commented on Oct 14, 2015

'oracle' is not a perfect name here, since when you google 'rustlang oracle', Google don't know what you intend to search, a compiler tool or a database api?

eddyb commented on Oct 14, 2015

I forgot to comment this on the pre-RFC thread (sorry @nrc), but I prefer rider as the name of the oracle tool, and keeping racer (with @phildawes' permission, of course) as the one-shot completion (and perhaps quickcheck) tool.

The way I see it, racer "races" to give an useful result each time, whereas rider "rides" along an IDE, for as long as there are Rust projects to be handled by it, and can have a slow start without impacting UX.

kud1ing commented on Oct 14, 2015

I don't have a name suggestion (yet), but i expect calling it "Oracle" will lead to confusion and legal trouble.

killercup commented on Oct 14, 2015

(I like how this RFC concentrates on the name first )

How about rustcage? Either pronounced rust cage or rust sage.

Personally I'm not entirely sold on the oracle/rider concept yet. I think the fundamental requirement is that we need a stable interface to rustc to support IDEs and plugins. I'm not yet sure whether a long running database process is required/desirable.

Some things that might cause problems:

• The oracle concept implies a single view of the sourcecode, and I think this jibes a bit with the tools that will perform refactoring, reformatting and suggestions. I suspect they will want interactive access to the compiler (compile this pre-processed snippet, what errors occur if I do this?)
• If the oracle were to support completion plugins directly from its database, it would need to have very low latency update turnaround (i.e. on every keypress, in <100ms). I'm not sure how well this would work with it being separate from rustc and not driven by the plugins.

It may be that we have an oracle in addition to a tools-oriented interface to rustc.

(aside: I noticed that the go oracle isn't used by the gocode completion tool, but I don't know the reason for this, it could just be historical)

Apologize for linking my own paper. [1] is a (proto-) pattern language describing methods to support multiple analyses over a changing source base (some from Smalltalk designs, some also used then in Eclipse). Basically, something like the oracle discussed here. One major decision point is: what objects from the analysis persist when the program text is not completely valid? for example, adding a "{" early in a file can be seen to invalidate every scope after it, do we then not use those function definitions in auto complete? Smalltalk solves this by giving most definitions (classes, methods) a persistent identity over time. The text edited only corresponds to a single definition, and updates the canonical version only when saved (and syntactically valid). Rust currently seems to encourage large files, which makes this more difficult, though not necessarily impossible. For example, if we have both a recent valid version of the source and the changed span, we can use the valid old definitions except where valid current versions are available.

[1] http://hillside.net/plop/2006/Papers/ACMConferenceProceedings/Intimacy_Gradient/a15-vainsencher.pdf

eddyb commented on Oct 15, 2015

@phildawes To remove as much latency as possible and to prevent rustc stabilization concerns, I believe that the oracle should use rustc's internal APIs and be behind the stability wall.
That is, the oracle ships with rustc and uses its libraries directly.

I really see no point in having some output format from rustc itself, that's added design complexity and inefficiency for no gain (in case of the oracle, at least).

Hi @eddyb!

I agree with that idea, if oracle is to be a thing then it makes sense to put oracle behind the stabilization wall and link it directly to rustc.

However I'm worried about the whole oracle thing. Definitely we need a stable interface to rustc. My concerns with oracle being the stable interface to rustc are:

• 'database of code' is seductive, but imposing this top-down abstraction could easily result in incidental complexity (overengineering)
• it might turn out that building the various refactorings and functionality require an interface that doesn't fit well into the database-of-code paradigm.

It already appears to me that completions maybe don't fit naturally into this design, and we've barely got started.

I'd prefer to see us drive the interface out bottom-up from the ide plugins. Try to build stuff using rustc directly, understand what stable interface is required.

nrc commented on Oct 15, 2015

'oracle' is not a perfect name here

@liigo oracle is a terrible name! But it does have some precedent from Go, and I can't think of a better one. Suggestions welcome!

nikomatsakis commented on Jan 23, 2016

Personally I'm wary of the database-of-code metaphor. I think it
hides the two hard problems of rust code completion, and instead
encourages thinking about indexing and cache invalidation.

This makes a lot of sense. I certainly agree that adding an oracle
doesn't get us fast code completion for free -- that seems to require
not just incr. comp., but also a bit more work towards lazy
compilation and better error recovery (some of which is ongoing, as
@nrc notes). It does seem like an oracle would help for Find All
References, and I'm not sure what else.

Still, it seems to me that even in code completion, there is perhaps a
role for an oracle. In particular, you talk about wanting to determine
the type of the receiver of a method, but once you know that receiver,
you still want to figure out things like "what are the set of traits
that are implemented for that type?" (and note that those traits may
not be imported, so the compiler might not consider them
normally). This might be an ideal place for an oracle to come in.

Moreover, some completion scenarios don't necessarily require knowing
the full type, e.g., if I write some_fn(, then you can likely
resolve some_fn to a specific item (but in fact you really want to
be resolving some_fn all along to potential matches, even before
I've finished typing it).

Maybe there's another way to say what I'm saying. It seems clear that
"ideal IDE integration" is going to be a long time coming, and we're
going to be evolving how rustc works for some time. But in the short
to medium time frame, I expect you are going to want to do fast,
approximate queries across several crates, and it seems like we can
get that more easily through some spearate oracle (you will also
want to be able to ask rustc for stuff and get back a fast answer, of
course).

And if in the future it happens that rustc gets fast enough that we
don't need a database to answer those queries, we can just recompute
the data, that's fine, right? All the old code that expected
potentially stale answers should be fine when their answers are 100%
up to date?

https://crates.io/crates/syntex_syntax/

Speaking of, who maintains this project? Are the Rust developers involved?

I think changes and improvements to the Rust parser would a nice incremental step and starting point to improving the toolchain, with regards to how it's used by IDEs or IDE related tools (like Racer).

I've been working on one such tool, https://github.com/RustDT/Rainicorn (formerly called rust-parse-describe, I mentioned in a previous comment some weeks ago), also using syntex_syntax, and I can see some avenues for improvement already. For starters a panic-less parser would be nice, the way the parser currently works makes my code a bit more complicated than would otherwise be necessary.

Forking a project is a socially aggressive action, and it is a shame to have political fallout and hurt over some small technical point or direction.

I didn't mean forking as in a divergent, or organizational fork (duplication of efforts, vastly incompatible code baselines, no communication, development heading in different directions, etc.).

I meant fork as in the Git way: a clone/branch that you create, you make some modifications or additions, but it doesn't diverge that much from the upstream source, and you try to merge updates from the upstream source regularly. And occasionally you might submit patches to upstream as well.

nrc commented on Jan 23, 2016

@bruno-medeiros syntex is maintained by @erickt, who is a core member of the community (community and moderation teams, as well as just being generally involved), but not part of the compiler team. It is pretty much just a straight clone of libsyntax from the compiler, so improvements to the compiler's parser show up in syntex quite quickly.

Part of the plan with procedural macros/syntax extensions is to present a stable interface for them to work on, at which point syntex gets a lot less necessary (only useful for tools). In the long term I'd like to stabilise enough of libsyntax that tools don't need it either.

There is work going on to make the parser panic-less, it no longer panics on error. I've also been doing some work on error recovery, for example rust-lang/rust#31065 adds some error correction for missing identifiers. Ideally it should be panic-free under normal use and recover from most errors within the next few months.

nrc commented on Jan 23, 2016

@nikomatsakis I did not intend the DB to be used for the code completion suggestions. It is useful for find all references, also queries like find all impls of a given trait, find all sub-traits, etc.

Still, it seems to me that even in code completion, there is perhaps a role for an oracle

Just to be clear, the Oracle, as in, "the resident process that is responsible for serving requests of various sorts to the IDE", it should handle code completion as well. Even if the data structures that are used to determine the results of code completion are entirely different from the data structures used for say, find-references, it makes no sense for this to be two different processes, or two different tools. This is because at the very minimum, these two operations can share cached AST data, not to mention that eventually (and sooner that later) we will want the Oracle to support functionality to manage/supply dirty editor buffers (ie, use a document that is being edited in memory in an IDE, but has not yet been persisted to disk). Even the functionality I'm coding in the Rainicorn tool should ideally also eventually be integrated into an oracle.

Of course, as an early prototype, it's okay for different operations to be handled by different tools, etc. but the end goal should be to integrate everything in the oracle, all-knowing that it is. (gotta hand it to the Go guys, they choose the perfect name - apart from the trademark thing.. )

@nrc BTW, I was just looking at the Nim language, and to my surprise found out they have an "oracle" tool already: http://nim-lang.org/docs/idetools.html
And by the looks of it quite more advanced than the Go-oracle, it actually seems to achieve all those key aspects that were mentioned before:

• Handles incorrect/incomplete code well
• Is a resident process
• Supports managing dirty buffers
• Supports all sorts of IDE query operations in a single tool: resolve-definition ("Definitions"), code-completion ("Suggestions"), find references ("Symbol usages"), etc.. (Something like parse-analysis seems to be missing though)

Ideally it should be panic-free under normal use and recover from most errors within the next few months.

Sweet

This RFC is now entering its two-week long final comment period

Don't know if it is of any help but I will throw it out there.
QtCreator(mainly C++ IDE) has a plugin that uses Clang to offer C++ code completion.
Here is the code:
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/libs/clangbackendipc
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/plugins/clangcodemodel

QtCreator also has an older custom C++ code model(that is being replaced by the clang one) that was a lot faster and seems like it is accepted in the community that the code model that uses the compiler will be considerably slower, but it offers more information and is more accurate.
I mention the speed because I saw @phildawes mentioning the hard requirements on the compiler to deliver the information in <100ms and I am not sure if we should expect it to be fast.

@lilianmoraru The reason the compiler needs to deliver the information so quickly is that the GUI is waiting on it. The user is expecting the completion to appear as soon as the user presses . and that requires the compiler to respond fast.

michaelwoerister commented on Feb 5, 2016

I'm in favor of accepting this RFC. The approach seems worth exploring and it does not preclude improving the compilers amenability for being used as a library. On the contrary, I think the compiler's APIs will benefit from trying to build the RLS on top of it and it can only help if people on the compiler team are actual clients of their own APIs. The RFC leaves many open questions when it comes to specifics and we just need a prototype implementation and the experience that comes from building that in order to decide how to proceed further. Worst case, we'll learn a bunch of stuff on what doesn't work :)

1-based line numbers and 0-based column numbers please.

nrc commented on Feb 9, 2016

@erkinalp why?

@erkinalp why?

I was wondering the same, why is a mixed format being used (1-based in one, and 0-based for the other)

Emacs uses 0-based column numbers.

ticki commented on Feb 10, 2016

@erkinalp That's a perfect reason for not doing that .

nrc commented on Feb 10, 2016

afaik, there is no standard for editors to use 1-based line numbers. Having both 0-based seems like the least confusing thing to do, it's easy enough for editors to add one to each line number.

dgrunwald commented on Feb 10, 2016

Compilers (including rustc) tend to use 1-based line and column numbers in their error messages. Following that standard seems like the least confusing thing to do. I certainly wouldn't expect 0-based line numbers.

But the concepts of "line" and "column" are ill-defined anyways.
Does a vertical tab (\v) count as a new line? What about form feed? What about all the other exotic Unicode newlines?
Does a tab count as N columns (where N is configurable; usually 4 or 8?), does it advance to the next multiple of N columns, or does it count as only 1 character?
Are full-width characters like 'ｘ' one or two columns wide?
Or does the editor count each Unicode codepoint as 1 column? Maybe a 'column' really is a UTF-8 byte offset within the line? Maybe it's measured in UTF-16 code units? Grapheme clusters?

It's difficult to find two editors that agree in their line/column counting for all possible input files.

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

it's easy enough for editors to add one to each line number.

But why should they? Line & column numbers coming from your oracle will be shown to the user and they expect 1-based numbering.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

Vertical tab means skip one line below and continue from same column offset.
CR, CR+LF, LF, LS and NEL are regular line feeds.
FF and PS count as two lines instead of one.

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

Why byte offsets and not Unicode character offsets? It's not like an error or position for a Rust symbol will ever start in the middle of a Unicode character.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

Because the internal API for lines and columns can be 0-based, despite the UI being 1-based. This is certainly the case for Eclipse, for IntelliJ, and probably for most IDEs/editors out there. It would not surprise me if Vim is the odd one out...

Keep it simple. Just use 0-based for lines and columns.

This RFC was discussed during the tools team triage today and the decision was to merge. This RFC is still at a somewhat high level and some minor details can continue to be ironed out in the implementation over time, but there seems to be widespread agreement about the body of the RFC here.

Thanks again for the discussion everybody!