| Table of Contents |
|---|
Useful projects
https://github.com/nicoulaj/idea-markdown
PSI
Parse trees versus PSI trees versus element types
[Note: The documentation and class hierarchy is a bit confusing because the root of all this stuff is ASTNode for both parse tree nodes and PSI nodes. The documentation calls the parse tree an AST, which is very misleading. It should strictly be parse tree or syntax tree. The PSI tree is actually the abstract syntax tree (AST) in typical literature. The ASTFactory seems to create parse tree internal nodes and leaves that can be psi or parse tree nodes. For example, the CoreASTFactory creates LeafPsiElement nodes by default.]
A lexer creates tokens (PsiBuilderImpl.Token) with specific token types, identified by IElementType. A parser creates a parse tree whose internal nodes are also identified by certain types that derive from IElementType as well. For example, consider assignment like "x = 3;" for which we might create IElementType instances for the identifier, =, the integer and the ;. Further, we might create a subclass
| Table of Contents |
|---|
Useful projects
https://github.com/nicoulaj/idea-markdown
PSI
Parse trees versus PSI trees versus element types
[Note: The documentation and class hierarchy is a bit confusing because the root of all this stuff is ASTNode for both parse tree nodes and PSI nodes. The documentation calls the parse tree an AST, which is very misleading. It should strictly be parse tree or syntax tree. The PSI tree is actually the abstract syntax tree (AST) in typical literature. The ASTFactory seems to create parse tree internal nodes and leaves that can be psi or parse tree nodes. For example, the CoreASTFactory creates LeafPsiElement nodes by default.]
A lexer creates tokens (PsiBuilderImpl.Token) with specific token types, identified by IElementType. A parser creates a parse tree whose internal nodes are also identified by certain types that derive from IElementType as well. For example, consider assignment like "x = 3;" for which we might create IElementType instances for the identifier, =, the integer and the ;. Further, we might create a subclass of IElementType called AssignmentNode to identify the syntactic structure (an internal parse tree node type). The parse tree or syntax trees are built for the entire file.
...
and got this error when I tried to backspace over the leftbracket:
...
| Code Block |
|---|
java.lang.IndexOutOfBoundsException: Wrong offset: 8. Should be in range: [0, 7] at com.intellij.openapi.editor.ex.util.SegmentArray.findSegmentIndex(SegmentArray.java:96) at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.documentChanged(LexerEditorHighlighter.java:227) at com.intellij.openapi.editor.impl.DocumentImpl.changedUpdate(DocumentImpl.java:592) at com.intellij.openapi.editor.impl.DocumentImpl.access$600(DocumentImpl.java:51) at com.intellij.openapi.editor.eximpl.utilDocumentImpl$MyCharArray.SegmentArray.findSegmentIndexafterChangedUpdate(SegmentArrayDocumentImpl.java:96881) at com.intellij.openapi.editor.eximpl.utilCharArray.LexerEditorHighlighter.documentChangedremove(LexerEditorHighlighterCharArray.java:227286) at com.intellij.openapi.editor.impl.DocumentImpl.changedUpdatedeleteString(DocumentImpl.java:592374) at com.intellij.openapi.editor.impl.DocumentImpl.access$600(DocumentImpl.java:51) at com.intellij.openapi.editor.impl.DocumentImpl$MyCharArray.afterChangedUpdate(DocumentImpl.java:881) at com.intellij.openapi.editor.impl.CharArray.remove(CharArray.java:286) at com.intellij.openapi.editor.impl.DocumentImpl.deleteString(DocumentImpl.java:374) at com.intellij.openapi.editor.actions.BackspaceAction.doBackSpaceAtCaret(BackspaceAction.java:104) ... |
It turns out that I had a lexical mode for [...] arg actions that did not take into consideration EOF before seeing the ]. The ANTLR lexer returned token:
[@-1,7:26='[grammar T;\n\na : ID;',<-1>,1:7]
where <-1> means EOF token type. So the intellij lexer decided that there was no more input or something after index 7. When I delete the [, all of a sudden, there is a bunch more input. The key is to always account for all of the input with the token set sent to the highlighter etc...
The dreaded "IntellijIdeaRulezzz" string
http://devnet.jetbrains.com/thread/454069?tstart=0
Colin Flemin: That string is inserted during completion, to ensure that there's always a valid identifier at the caret when running completion. You can customise it to whatever would be a valid identifier in your language, but generally you need something (i.e. you can't customise it to insert an empty string, as Sam tried). The way this works is during completion the PSI is copied and that symbol is inserted into the copy. You then do your autocomplete logic but that symbol is never inserted into your original document. The only way it sneaks into the real document is generally bugs in the autocomplete code, i.e. creating an autocomplete item from the dummy symbol.
The token is inserted to make autocomplete logic more consistent - often autocomplete needs to take into account the surrounding context and without an identifier there that context is often invalid. Imagine that I'm completing a static method call in Java, say Integer.parseInt(). If I have the Integer.| (with | representing the cursor) that's going to be an invalid chunk of code which would complicate the completion - having Integer.IntelliJRulezz at least makes the immediate context for completion a valid identifier, even though it's a little nonsensical.
Jon Acktar: The token is inserted into a copy of the file, so your lexer gets to lex your original and the copy at the same time, since you have probably typed something in the original document the original gets reprocessed by the lexer along with the copy that has been made. So during completion there are 2 copies of the file, one with the string and one without.
Maybe my solution is to simply ignore errors from a lexer working on the copy? For my particular problem at the moment, I'm building an input window for the ANTLR plug-in, for which I do not need auto completion so I can simply ignore that lexer maybe. This behavior would explain why I saw more lexing requests than I thought were needed.
It looks like the second lexer/parser is invoked from CompletionAutoPopupHandler.invokeCompletion() when there is that strange string on there. I am altering my parser definition object to return an empty lexer and empty parser when I detect that string.That did not work. You have to make the lexer skip the strange string first. The character input stream sets the end offset to the start of that dummy string. That way the lexer never sees it. Two parsers are created, one for the auto completion process and it contains the dummy string. All we have to do is avoid setting the visual parse tree to that particular parse tree. a parse errors that come from that we don't care about. I have no idea why it is calling the auto complete when I have all that turned off.
I gave up on this and simply spent a few hours getting my own error annotations added to a standard editor window based upon ANTLR parsing. Much easier than figuring out the quirky and finicky parsing infrastructure within intellij.
Turning off auto completion lexing
In the end it became clear that we cannot turn this off because other plug-ins might need it. I'm not sure this will work anyway.
Peter Gromov: You can turn autopopup completion off by providing a CompletionConfidence extension (with order="first"). You can also change the dummy identifier to something your lexer can handle (maybe even empty): com.intellij.codeInsight.completion.CompletionInitializationContext#setDummyIdentifier, com.intellij.codeInsight.completion.CompletionContributor#beforeCompletion
Custom Structure view.
Hook in a StructureViewBuilderProvider via plugin.xml:
<structureViewBuilder factoryClass="org.antlr.jetbrains.st4plugin.structview.STGroupStructureViewBuilderProvider"/>
...
.intellij.openapi.editor.actions.BackspaceAction.doBackSpaceAtCaret(BackspaceAction.java:104)
... |
It turns out that I had a lexical mode for [...] arg actions that did not take into consideration EOF before seeing the ]. The ANTLR lexer returned token:
[@-1,7:26='[grammar T;\n\na : ID;',<-1>,1:7]
where <-1> means EOF token type. So the intellij lexer decided that there was no more input or something after index 7. When I delete the [, all of a sudden, there is a bunch more input. The key is to always account for all of the input with the token set sent to the highlighter etc...
The dreaded "IntellijIdeaRulezzz" string
http://devnet.jetbrains.com/thread/454069?tstart=0
Colin Flemin: That string is inserted during completion, to ensure that there's always a valid identifier at the caret when running completion. You can customise it to whatever would be a valid identifier in your language, but generally you need something (i.e. you can't customise it to insert an empty string, as Sam tried). The way this works is during completion the PSI is copied and that symbol is inserted into the copy. You then do your autocomplete logic but that symbol is never inserted into your original document. The only way it sneaks into the real document is generally bugs in the autocomplete code, i.e. creating an autocomplete item from the dummy symbol.
The token is inserted to make autocomplete logic more consistent - often autocomplete needs to take into account the surrounding context and without an identifier there that context is often invalid. Imagine that I'm completing a static method call in Java, say Integer.parseInt(). If I have the Integer.| (with | representing the cursor) that's going to be an invalid chunk of code which would complicate the completion - having Integer.IntelliJRulezz at least makes the immediate context for completion a valid identifier, even though it's a little nonsensical.
Jon Acktar: The token is inserted into a copy of the file, so your lexer gets to lex your original and the copy at the same time, since you have probably typed something in the original document the original gets reprocessed by the lexer along with the copy that has been made. So during completion there are 2 copies of the file, one with the string and one without.
Maybe my solution is to simply ignore errors from a lexer working on the copy? For my particular problem at the moment, I'm building an input window for the ANTLR plug-in, for which I do not need auto completion so I can simply ignore that lexer maybe. This behavior would explain why I saw more lexing requests than I thought were needed.
It looks like the second lexer/parser is invoked from CompletionAutoPopupHandler.invokeCompletion() when there is that strange string on there. I am altering my parser definition object to return an empty lexer and empty parser when I detect that string.That did not work. You have to make the lexer skip the strange string first. The character input stream sets the end offset to the start of that dummy string. That way the lexer never sees it. Two parsers are created, one for the auto completion process and it contains the dummy string. All we have to do is avoid setting the visual parse tree to that particular parse tree. a parse errors that come from that we don't care about. I have no idea why it is calling the auto complete when I have all that turned off.
I gave up on this and simply spent a few hours getting my own error annotations added to a standard editor window based upon ANTLR parsing. Much easier than figuring out the quirky and finicky parsing infrastructure within intellij.
Turning off auto completion lexing
In the end it became clear that we cannot turn this off because other plug-ins might need it. I'm not sure this will work anyway.
Peter Gromov: You can turn autopopup completion off by providing a CompletionConfidence extension (with order="first"). You can also change the dummy identifier to something your lexer can handle (maybe even empty): com.intellij.codeInsight.completion.CompletionInitializationContext#setDummyIdentifier, com.intellij.codeInsight.completion.CompletionContributor#beforeCompletion
Custom Structure view.
Hook in a StructureViewBuilderProvider via plugin.xml:
<structureViewBuilder factoryClass="org.antlr.jetbrains.st4plugin.structview.STGroupStructureViewBuilderProvider"/>
See https://devnet.jetbrains.com/message/5564298 and https://devnet.jetbrains.com/thread/475435?tstart=0
public class STGroupStructureViewBuilderProvider implements StructureViewBuilderProvider {
@Nullable
@Override
public StructureViewBuilder getStructureViewBuilder(@NotNull FileType fileType, @NotNull VirtualFile file, @NotNull Project project) {
return new STGroupStructureViewBuilder();
}
}
public class STGroupStructureViewBuilder implements StructureViewBuilder {
@NotNull
@Override
public StructureView createStructureView(FileEditor fileEditor, Project project) {
return new STGroupStructureView();
}
}
public class STGroupStructureView implements StructureView {
@Override
public void centerSelectedRow() {
}
@Override
public FileEditor getFileEditor() {
return null;
}
...
}public class STGroupStructureViewTreeElement
implements StructureViewTreeElement, ItemPresentation
{
@Nullable
@Override
public Icon getIcon(boolean unused) {
return null;
}
@Nullable
@Override
public StructureViewBuilderString getStructureViewBuilder(@NotNull FileType fileType, @NotNull VirtualFile file, @NotNull Project projectgetPresentableText() {
return new STGroupStructureViewBuilder()null;
}
...
}
You also need to invalidate tree upon editor changes
/** Invalidate tree upon doc change */
public classvoid STGroupStructureViewBuilderregisterStructureViewModel(final implementsEditor StructureViewBuildereditor, {
final STGroupStructureViewModel model) @NotNull{
@Override
public StructureViewfinal createStructureView(FileEditor fileEditor, Project project) {Document doc = editor.getDocument();
final DocumentListener listener return= new STGroupStructureViewDocumentAdapter();
}
}public class STGroupStructureView implements StructureView {
{
@Override
public void centerSelectedRowdocumentChanged(DocumentEvent e) { model.invalidate(); }
};
DocumentListener @Override
public FileEditor getFileEditor() {oldListener = doc.getUserData(EDITOR_STRUCTVIEW_LISTENER_KEY);
if ( oldListener!=null return) null;{
}
...
}public class STGroupStructureViewTreeElementdoc.removeDocumentListener(oldListener);
}
implements StructureViewTreeElement, ItemPresentation
{doc.putUserData(EDITOR_STRUCTVIEW_LISTENER_KEY, listener);
@Nullable
@Override
public Icon getIcon(boolean unused) {
return null;
}
@Nullable
@Override
public String getPresentableTextdoc.addDocumentListener(listener);
}
In the view model, we set
/** If parseTree==null, this will return a StructureViewTreeElement with
* getValue()==null, which forces rebuild in {@link com.intellij.ide.impl.StructureViewWrapperImpl#rebuild()}
*/
@NotNull
@Override
public StructureViewTreeElement getRoot() {
return return null;
}
...
new STGroupRootStructureViewTreeElement(parseTree,file);
}
Resolving references
All PsiElement can getReference() and getReferences() which return PsiReference. That reference can really be anything we want. I see PsiBreakStatementImpl returning a PsiLabelReference that wraps the break statement and the identifier in the break statement.
...
http://confluence.jetbrains.com/display/IntelliJIDEA/Annotator
Annotator helps highlight and annotate any code based on specific rules.
Doc:
The third level of highlighting is performed through the Annotator interface. A plugin can register one or more annotators in the com.intellij.annotator extension point, and these annotators are called during the background highlighting pass to process the elements in the PSI tree of the custom language. Annotators can analyze not only the syntax, but also the semantics of the text in the language, and thus can provide much more complex syntax and error highlighting logic. The annotator can also provide quick fixes to problems it detects.
...
final JList list = new JBList(stack);
// PopupChooserBuilder builder = new PopupChooserBuilder(list);
JBPopupFactory factory = JBPopupFactory.getInstance();
PopupChooserBuilder builder = factory.createListPopupBuilder(list);
JBPopup popup = builder.createPopup();
MouseEvent mouseEvent = event.getMouseEvent();
Point point = mouseEvent.getPoint();
Dimension dimension = popup.getContent().getLayout().preferredLayoutSize(builder.getScrollPane());
System.out.println(dimension);
int height = dimension.height;
point.translate(10, -height);
RelativePoint where = new RelativePoint(mouseEvent.getComponent(), point);
popup.show(where);
Code for a balloon.
JBPopupFactory popupFactory = JBPopupFactory.getInstance();
BalloonBuilder builder =
popupFactory.createHtmlTextBalloonBuilder(Utils.join(stack.toArray(), "<br>"),
MessageType.INFO, null);
builder.setHideOnClickOutside(true);
Balloon balloon = builder.createBalloon();
MouseEvent mouseEvent = event.getMouseEvent();
Point point = mouseEvent.getPoint();
point.translate(10, -15);
RelativePoint where = new RelativePoint(mouseEvent.getComponent(), point);
balloon.show(where, Balloon.Position.above);
...